How To Wait For Page Load To Complete?

I'm trying to get available boot size (under $('option.addedOption')) from http://www.neimanmarcus.com/Stuart-Weitzman-Reserve-Suede-Over-the-Knee-Boot-Black/prod179890262/p.prod I

Solution 1:

As I understand correctly: no matter how long the code sleeps it still hasn't loaded the shoe size?

Since you are not using a headless browser you do not execute javascript on the requested page. Try using a headless browser like PhantomJS. Here a list of more headless browsers.

Here one way how to use PhantomJS in Python.

Solution 2:

Use it like:

with urllib2.urlopen(request) as response:
   html = response.read()
   print html
   html = fromstring(html)
   sel = CSSSelector('option.addedOption')
   try:
       options = sel(html)
       print options
   except Exception as e:
       print e

instead of

html = urllib2.urlopen(request)
time.sleep(10)
html = html.read()
...