Python Subprocess Batch Call

I am trying to create a variable batch size, to make a subprocess call. I am a bit confused on the best way to have a batch of 5 kick off, wait for all 5 to complete than kick off

Solution 1:

I think you're probably looking for something along these lines:

batchSize = 5
processArray = process.split(",")
for i in xrange(0, len(processArray), batchSize):
    batch = processArray[i:i+batchSize]  # len(batch) <= batchSize
    ps = []
    for process in batch:
        output = "..."
        p = subprocess.Popen(process + ">>" + output, shell=True, stdout=subprocess.PIPE)
        ps.append(p)
    for p in ps:
        p.wait()

Solution 2:

You want to do something like this. Say you have a list, commands, of all the commands you want to run.

defchunks(l, n):
    """Yield successive n-sized chunks from l."""for i in xrange(0, len(l), n):
        yield l[i:i+n]

for next_batch in chunks(commands, 5):
    # Start the next few subprocesses
    subps = [subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
             for cmd in next_batch]
    # Wait for these to finishfor subp in subps:
        subp.wait()

(Chunks function taken from this answer.)

Solution 3:

You need .communicate() or .wait() functions of subprocess module to wait for the processes to finish. Alternatively, you can use .poll() to see if a subprocess has finished.

batchSize = 5
proccessArray = process.split(",")
processLength = len(proccessArray) - 1
counter1 = batchSize
for i inrange(0, processLength, batchSize):
    d = {}
    for x inrange(1, batchSize):
        dgclOutput = inputPath + st + "_" + (i + x) + "output"
        d["process{0}".format(x)] = subprocess.Popen(proccessArray(i + x) + ">>" + dgclOutput, shell=True, stdout=subprocess.PIPE)
    whilenot counter1:
        for p in d:
            ifnot p.poll():
               counter1 -= 1

There is a better example here: Python subprocess in parallel