Python Scraping Fb Comments From A Website

I have been trying to scrape facebook comments using Beautiful Soup on the below website pages. import BeautifulSoup import urllib2 import re url = 'http://techcrunch.com/2012/05/

Solution 1:

Like Christopher and Thiefmaster: it is all because of javascript.

But, if you really need that information, you can still retrieve it thanks to Selenium on http://seleniumhq.org then use beautifulsoup on this output.

Solution 2:

Facebook comments are loaded dynamically using AJAX. You can scrape the original page to retrieve this:

<fb:commentshref="http://techcrunch.com/2012/05/15/facebook-lightbox/"num_posts="25"width="630"></fb:comments>

After that you need to send a request to some Facebook API that will give you the comments for the URL in that tag.

Solution 3:

The parts of the page you are looking for are not included in the source file. Use a browser and you can see this for yourself by opening the page source.

You will need to use something like pywebkitgtk to have the javascript executed before passing the document to BeautifulSoup