How To Bypass A 'cookiewall' When Using Scrapy?
I'm a new user to Scrapy. After following the tutorials for extracting data from websites, I am trying to accomplish something similar on forums. What I want is to extract all pos
Solution 1:
In the end I found multiple ways to solve this problem:
- Simply having adding
/?token=77c1f767bc31859fee1ffe041343fa48&allowcookies=ACCEPTEER+ALLE+COOKIES
to the start url worked for this specific case - I later switched to a
CrawlSpider
instead of a normal Spider, then I could add the xpath of the cookie button as the first rule. - Clicking the button using the earlier mentioned
Selenium
also worked, but is a lot of hassle that is not really necessary...
Post a Comment for "How To Bypass A 'cookiewall' When Using Scrapy?"