Re-using Deferred Objects In Twisted

In Twisted, it seems that a deferred object can only be used once after its callback has fired, as opposed to other 'promise'-based libraries I've worked with: from twisted.interne

Solution 1:

The issue is not that the Deferred itself can only be "used once" - it's infinitely re-usable, in the sense that you can keep adding callbacks to it forever and data will continue flowing to the next callback, and the next, as it's available. The problem you're seeing is that when you add a callback to a Deferred, its result is propagated to the next callback.

The other, intersecting problem here is that yielding a Deferred from an inlineCallbacks function is assumed to "consume" a Deferred - you're getting its value and doing something with it, so to prevent unnecessary resource utilization (that Deferred carrying around a result for longer than it needs to), the callback that gives you the result from the yield expression also itself returns None. It might be a little easier to understand if it returned some kind of more explicit "consumed by inlineCallbacks token", I suppose, but hindsight is 20/20 :-).

But in a sense, a Deferredcan only be "used" once, which is to say, if you have an API which returns a Deferred, it should return a newDeferred to each caller. By returning it, you're really transferring ownership to the caller, because callers may modify the result, to pass on to their own callers. The typical example is that if you have an API that returns a Deferred that fires with some bytes, but you know the bytes are supposed to be JSON, you might add .addCallback(json.loads) and then return it, which would allow that caller to consume the JSON-serialized object rather than the bytes.

So if you intend for async to be called multiple times, the way you would do it is something like this:

from __future__ import print_function, unicode_literals

from twisted.internet import defer

classFoo(object):

    def__init__(self):
        self.dfd = defer.Deferred()

    defasync(self):
        justForThisCall = defer.Deferred()
        defcallbackForDFD(result):
            justForThisCall.callback(result)
            return result
        self.dfd.addCallback(callbackForDFD)
        return justForThisCall

    @defer.inlineCallbacksdeffunc(self):
        print('Started!')
        result = yield self.async()
        print('Stopped with result: {0}'.format(result))

if __name__ == '__main__':
    foo = Foo()
    print("calling func")
    foo.func()
    print("firing dfd")
    foo.dfd.callback('no need to wait!')
    print("calling func again")
    foo.func()
    print("done")

which should produce this output:

calling func
Started!
firing dfd
Stopped with result: no need to wait!
calling func again
Started!
Stopped with result: no need to wait!
done