How To Increase Scrapy Crawling Speed?

I am using Scrapy to crawl websites and extract data to a json file, but I've found that for some sites the crawler takes ages to crawl the complete website. My question is: How ca

Solution 1:

Try tuning the CONCURRENT_ITEMS, CONCURRENT_REQUESTS, CONCURRENT_REQUESTS_PER_DOMAIN and other settings.

For full list of settings, see http://doc.scrapy.org/en/latest/topics/settings.html