Generating Url For Yahoo And Bing Scraping For Multiple Pages With Python And Beautifulsoup

I want to scrape news from different sources. I found a way to generate URL for scraping multiple pages from google, but I think that there is a way to generate much shorter link.

Solution 1:

I am not sure are you looking after this shorten url for news.

from bs4 import BeautifulSoup
import requests

headers = {'User-Agent':'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36'}

term = 'usa'
page=0for page inrange(1,5):

    page = page*10

    url = 'https://www.google.com/search?q={}&tbm=nws&start={}'.format(term,page)
    print(url)

    response = requests.get(url, headers=headers,verify=False)
    soup = BeautifulSoup(response.text, 'html.parser')

#Yahoo:

from bs4 import BeautifulSoup
import requests

headers = {'User-Agent':'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36'}

term = 'usa'
page=1whileTrue:

    url ='https://news.search.yahoo.com/search?q={}&pz=10&b={}'.format(term,page)
    print(url)
    page = page + 10
    response = requests.get(url, headers=headers,verify=False)
    if response.status_code !=200:
        break
    soup = BeautifulSoup(response.text, 'html.parser')