Skip to content Skip to sidebar Skip to footer

How To Bypass A 'cookiewall' When Using Scrapy?

I'm a new user to Scrapy. After following the tutorials for extracting data from websites, I am trying to accomplish something similar on forums. What I want is to extract all pos

Solution 1:

In the end I found multiple ways to solve this problem:

  • Simply having adding /?token=77c1f767bc31859fee1ffe041343fa48&allowcookies=ACCEPTEER+ALLE+COOKIES to the start url worked for this specific case
  • I later switched to a CrawlSpider instead of a normal Spider, then I could add the xpath of the cookie button as the first rule.
  • Clicking the button using the earlier mentioned Selenium also worked, but is a lot of hassle that is not really necessary...

Post a Comment for "How To Bypass A 'cookiewall' When Using Scrapy?"