Get Stdout, Stderr Without Waiting For Process To Exit

Some processes exit after a long time and their status is constantly being written to STDOUT. How can I run a process and read its STDOUT without waiting for the process to exit? I

Solution 1:

Reading line by line is straight forward

p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)
for line in p.stdout:
    do work here
p.wait()
if p.returncode != 0:
    panic here

...mostly. Commands tend to buffer differently depending on whether they are run from a conole or another program. Since you are not a console, you may find that it outputs less often... but it'll all eventually get there before the program exits. On linux you can use pty or pexpect or something, but I know of no good solution on windows.

Now if you want to run them in parallel, create a thread per command you run and thread.join() each of them at the end of your programming. This is a bit tricky in itself. You could use a thread pool (see multiprocessing.ThreadPool on python 2.x, don't remember the name off hand in 3.x).