Hitting Multiple Apis At Once, Tornado And Python

I'm trying to make an API that will collect responses from several other API's and combine the results into one response. I want to sent the get requests asynchronously so that it

Solution 1:

There's many reasons why your code doesn't work. To begin, requests generally blocks the event loop and doesn't let anything else execute. Replace requests with AsyncHTTPClient.fetch. Also, the way you were yielding each request would also make the requests sequential and not concurrently like you thought. Here's an example of how your code could be restructured:

import json
from tornado import gen, httpclient, ioloop, web

# ...classMyApi(web.RequestHandler):

    @gen.coroutinedefget(self):
        futures_list = []
        for path in PATHS:
            futures_list.append(self.path_get(path))

        yield futures_list
        result = json.dumps({'results': [x.result() for x in futures_list]})
        self.write(result)

    @gen.coroutinedefpath_get(self, path):
        request = httpclient.AsyncHTTPClient()
        resp = yield request.fetch(path)
        result = json.loads(resp.body.decode('utf-8'))
        raise gen.Return(result)

What's happening is we're creating a list of Futures that get returned from gen.coroutine functions and yielding the entire list until the results from the request are available. Then once all the requests are complete, futures_list is iterated and the results are used to create a new list which is appended to a JSON object.