PythonAnywhere Forums

scrapy basic application


today i've updated my account to a payed one, just to test out mi newborn application on scrapy framework...

but when i try to scrape the basic example host "" the warning remain the same:

2017-06-18 15:16:13 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)

My application is pretty simple (is the example tutorial):

# -*- coding: utf-8 -*-
import scrapy

class ToScrapeCSSSpider(scrapy.Spider):
    name = "toscrape-css"
    start_urls = [

    def parse(self, response):
        for quote in response.css("div.quote"):
            yield {
                'text': quote.css("span.text::text").extract_first(),
                'author': quote.css("").extract_first(),
                'tags': quote.css("div.tags > a.tag::text").extract()

        next_page_url = response.css(" > a::attr(href)").extract_first()
        if next_page_url is not None:
            yield scrapy.Request(response.urljoin(next_page_url))

maybe i've missed something ??

Nevermind, now it works, it take a while to unlock the whitelist i suppose...

glad you got it working! :)