Python Subprocess Batch Call
I am trying to create a variable batch size, to make a subprocess call. I am a bit confused on the best way to have a batch of 5 kick off, wait for all 5 to complete than kick off
Solution 1:
I think you're probably looking for something along these lines:
batchSize = 5
processArray = process.split(",")
for i in xrange(0, len(processArray), batchSize):
batch = processArray[i:i+batchSize] # len(batch) <= batchSize
ps = []
for process in batch:
output = "..."
p = subprocess.Popen(process + ">>" + output, shell=True, stdout=subprocess.PIPE)
ps.append(p)
for p in ps:
p.wait()
Solution 2:
You want to do something like this. Say you have a list, commands
, of all the commands you want to run.
defchunks(l, n):
"""Yield successive n-sized chunks from l."""for i in xrange(0, len(l), n):
yield l[i:i+n]
for next_batch in chunks(commands, 5):
# Start the next few subprocesses
subps = [subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
for cmd in next_batch]
# Wait for these to finishfor subp in subps:
subp.wait()
(Chunks function taken from this answer.)
Solution 3:
You need .communicate()
or .wait()
functions of subprocess
module to wait for the processes to finish. Alternatively, you can use .poll()
to see if a subprocess has finished.
batchSize = 5
proccessArray = process.split(",")
processLength = len(proccessArray) - 1
counter1 = batchSize
for i inrange(0, processLength, batchSize):
d = {}
for x inrange(1, batchSize):
dgclOutput = inputPath + st + "_" + (i + x) + "output"
d["process{0}".format(x)] = subprocess.Popen(proccessArray(i + x) + ">>" + dgclOutput, shell=True, stdout=subprocess.PIPE)
whilenot counter1:
for p in d:
ifnot p.poll():
counter1 -= 1
There is a better example here: Python subprocess in parallel
Post a Comment for "Python Subprocess Batch Call"