Non-blocking Generator On Python

I'm using a generator function from the requests module in a QT-Application, pretty much the same as in the requests-streaming example: import json import requests def get_stream(

Solution 1:

Take a look at the producer-consumer pattern. It is commonly implemented in python using a Queue.

The producer, typically running in a thread or another process (Queue supports either), simply puts messages in the queue. The consumer, whenever it feels like it, pops messages from the queue. This operation supports a timeout argument.

Solution 2:

As Simeon is asking in the comment, it cannot work as simple as you describe it in your example. There are quite some details you have to take care of. There are different solutions, which make more or less sense, depending on your use case. You don't give much details about what you really want to do, so I will just send you to http://twistedmatrix.com/trac/wiki/QTReactor as an example. There a different solutions/frameworks which implement async message queues. And I think, that's what you are looking for.

Solution 3:

If you control the generator function, one solution would be to have it throw an exception after a timeout period. Perhaps something along the lines of:

defget_stream(timeout=None):
    while message=read_message(timeout=timout):
        yield message

Then have read_message throw a TimeOutException or something if a timeout condition occurs.

Of course, you still have to deal with the logistics of when/how to retry/resume.

Solution 4:

You could use async generators as of python 3.6 https://www.python.org/dev/peps/pep-0525/

import json
import requests

asyncdefget_stream():
    r = requests.get('http://httpbin.org/stream/20', stream=True)
    for line in r.iter_lines():
        if line:
            yield json.loads(line)

asyncdefconsume_stream():
   awaitfor message in get_stream():
       #do something