Python: Multiprocessing Map Takes Longer To Complete Last Few Processes

In Python, I'm trying to run 150-200 processes. I have these 150 things in an array, and I've split this array up into multiple arrays of 10 elements each. Now, I run a Multiproces

Solution 1:

This is due to the way Pool.map distributes the data among the workers. Use chunksize=1 as a parameter, i.e. map(...,..., chunksize=1).

A similar problem is explained in: python multiprocessing map mishandling of last processes