Crawlspider Seems Not To Follow Rule

here's my code. Actually I followed the example in 'Recursively Scraping Web Pages With Scrapy' and it seems I have included a mistake somewhere. Can someone help me find it, plea

Solution 1:

I didn't find your code on the link you passed, but I think the problem is that you are never using the rules.

Scrapy starts crawling by calling the start_requests method, but the rules are compiled and used on the parse method, which you are not using because your requests go directly from start_requests to parse_items.

You could remove the callback on the start_requests method if you want the rules to be applied on that level.