Reusing Httplib.httpconnection In Python 2.7

I recently inherited a python project, and I'm working on maintaining it now. Part of the code makes a few hundred thousand requests from a website and saves the results to a data

Solution 1:

I think you first need to decide the failure mode you want to handle. For instance, did the connection reset because of a temporary resource problem on the server and a quick turnaround connect will fix it? Or, is the server down or rebooting and you should abort your process?

Presuming the first case, I think you are thinking along the right lines. Try something like this (note, this is not working code - it's just an example of the logic):

while True:
    try:
        conn.request("GET",someString,'',headers)
        response = conn.getresponse()
    except httplib.HTTPException, e:
        conn.connect()
        continuebreak

You should probably add some logic to that to pause between repeated connect attempts and to give up after a certain number of tries (which is basically the second scenario above).

In order to test this, try using tcpkill to cause the TCP connection to reset:

http://www.gnutoolbox.com/tcpkill-command/