Scraping A Webpage That Has Javascript With Beautifulsoup

guys! I am applying to you once again. I am ok with scraping simple websites with tags but recently I've encountered a quite complex website which has JavaScript. As a result I wou

Solution 1:

Looks like the data you're trying to extract is in a data model, which means it's in JSON. If you do a small amount of parsing with the following:

import json
import re

data_string = soup.findAll('script')[11].string.encode('utf8')
data_string = data_string.split("DataModel.parse(")[1]
data_string = data_string.split(");")[0]

// parse out erroneous html
while re.search('\<[^\>]*\>', datastring):
    data_string = ''.join(datastring.split(re.search('\<[^\>]*\>', datastring).group(0)))

// parse out other function parameters, leaving you with the json
data_you_want = json.loads(data_string.split(re.search('\}[^",\}\]]+,', data_string).group(0))[0]+'}')

print(data_you_want["estimate"])
>>> {'shares': {'shares_hash': {'twitter': None, 'stocktwits': None, 'linkedin': None}}, 'lastRevised': None, 'id': None, 'revenue_points': None, 'sector': 'financials', 'persisted': False, 'points': None, 'instrumentSlug': 'jpm', 'wallstreetRevenue': 23972, 'revenue': 23972, 'createdAt': None, 'username': None, 'isBlind': False, 'releaseSlug': 'fq3-2016', 'statement': '', 'errorRanges': {'revenue': {'low': 21247.3532016398, 'high': 26820.423240734}, 'eps': {'low': 1.02460526459765, 'high': 1.81359679579922}}, 'eps_points': None, 'rank': None, 'instrumentId': 981, 'eps': 1.4, 'season': '2016-fall', 'releaseId': 52773}

The DataModel.parse is a javascript method which means it ends with a parenthesis and a colon. the parameter for the function is the JSON object you want. By loading it into json.loads you're able to access it much like a dictionary.

From there you remap the data into the form you want it to be in for the csv.

Solution 2:

This is how I've resolved the problem, using some tips above:

from bs4 import BeautifulSoup
from urllib import urlopen
import json
import csv

f = csv.writer(open("estimize.csv", "a"))
f.writerow(["User Name", "Revenue Estimate", "EPS Estimate"])

html = "https://www.estimize.com/jpm/fq3-2016?sort=rank&direction=asc&estimates_per_page=142&show_confirm=false"
html = urlopen(html)
soup = BeautifulSoup(html.read(), "html.parser").encode('utf8')
data_string = soup.split("\"allEstimateRows\":")[1]
data_string = data_string.split(",\"tableSortDirection")[0]
data = json.loads(data_string)

for item indata:
    f.writerow([item["userName"], item["revenue"], item["eps"]])