Python - Same Xpath In Selenium And Lxml Different Results

I have this site http://www.google-proxy.net/ and i need to get first proxy's ip:port. br = webdriver.Firefox() br.get('http://www.google-proxy.net/') ip = br.find_element_by_xpat

Solution 1:

Looks like 'odd' classes are added by Javascript in this site.

Selenium, as it runs the browser, executes the Javascript, so you have the expected class.

requests library will not execute JS, so there's no 'odd' class.

Solution 2:

When you use Selenium to open http://www.google-proxy.net, JavaScript is enabled. In this case, JavaScript adds the classes odd and even to the tr elements.

The requests.get method loads the HTML from http://www.google-proxy.netwithout JavaScript enabled. So the classes odd and even are not added to the tr elements, and your XPath/lxml functionality doesn't select anything. To replicate this behaviour you can use JavaScript switcher plugins eg Chrome plugin. This allows you to easily load webpages without JavaScript enabled.