The Python Oracle

Pass kwargs to starmap while using Pool in Python

Become part of the top 3% of the developers by applying to Toptal https://topt.al/25cXVn

--

Music by Eric Matyas
https://www.soundimage.org
Track title: Magic Ocean Looping

--

Chapters
00:00 Question
01:03 Accepted answer (Score 15)
01:34 Thank you

--

Full question
https://stackoverflow.com/questions/4571...

--

Content licensed under CC BY-SA
https://meta.stackexchange.com/help/lice...

--

Tags
#python #multithreading #threadpool #pythonmultiprocessing

#avk47



ACCEPTED ANSWER

Score 23


You can create a wrapper around pool.starmap that also accepts an iterator of over kwargs dictionaries.

from itertools import repeat

def starmap_with_kwargs(pool, fn, args_iter, kwargs_iter):
    args_for_starmap = zip(repeat(fn), args_iter, kwargs_iter)
    return pool.starmap(apply_args_and_kwargs, args_for_starmap)

def apply_args_and_kwargs(fn, args, kwargs):
    return fn(*args, **kwargs)

Then you can call it in your case as:

args_iter = zip(repeat(project_name), api_extensions)
kwargs_iter = repeat(dict(payload={'a': 1}, key=True))
branches = starmap_with_kwargs(pool, fetch_api, args_iter, kwargs_iter)