Elements On Page Don't Exist When Scraping Wsj.com

I am using Python to scrape a webpage. This is my code: import requests from bs4 import BeautifulSoup # Set local variables URL = 'https://www.wsj.com/market-data/bonds' page = r

Solution 1:

You need to add the user-agent header, otherwise the page thinks that you’re a bot and will block you. Also note you had an extra space in your class name

import requests
from bs4 import BeautifulSoup

URL = 'https://www.wsj.com/market-data/bonds'


HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36"
}

page = requests.get(URL, headers=HEADERS)
soup = BeautifulSoup(page.content, 'html.parser')

table = soup.find("table", attrs={"class": "WSJTables--table--1QzSOCfq"})
print(table)