How To Use Non-top-level Functions In Parallelization?

I'd like to use multiprocessing in a rescource-heavy computation in a code I write, as shown in this watered-down example: import numpy as np import multiprocessing as multiproc d

Solution 1:

This probably isn't the best answer for this, but it's an answer, so please no hate :)

You can just write a top level wrapper function that can be serialized and have it execute functions... This is kinda like function inception a bit but I solved a similar problem in my code like this.

Here is a brief example

def wrapper(arg_list, *args):
    func_str = arg_list[0]
    args = arg_list[1]
    code = marshal.loads(base64.b64decode(func_str.data))
    func = types.FunctionType(code, globals(), "wrapped_func")
    return func(*args)

def run_func(func, *args):
    func_str = base64.b64encode(marshal.dumps(func.__code__, 0))
    arg_list = [func_str, args]
    with mp.Pool(2) as pool:
        results = pool.map(wrapper, arg_list)
    return results