Python Requests Module Doesn't Return Full Page During Get Request

When I make a get request to this url: http://www.waterwaysguide.org.au/waterwaysguide/access-point/4980/partial with a browser a full html page is returned. However when I make a

Solution 1:

The following approach displays the missing content inside the div class="view view-waterway-access-point-page...

>>>from urllib.request import Request, urlopen>>>from bs4 import BeautifulSoup>>>url = 'http://www.waterwaysguide.org.au/waterwaysguide/access-
point/4980/partial'
>>>req = Request(url,headers={'User-Agent': 'Mozilla/5.0'})>>>webpage = urlopen(req).read()>>>print(webpage)

Solution 2:

I found the error that I had made. I never used the 'point_num' argument that I pass to the function so my request was not going to the correct url.

The code is working now that I have changed the line to

r = requests.get(base_url.format(point_num))

Solution 3:

It might be the case that the elements are rendered using javascript AFTER the page has loaded. So, you only get the page and not the javascript rendered parts. You might want to look into https://medium.com/@hoppy/how-to-test-or-scrape-javascript-rendered-websites-with-python-selenium-a-beginner-step-by-c137892216aaWeb-scraping JavaScript page with Python