Problem with running the code in parallel

[MohamedNedal]

Hello, I was running the code as a Python script with lots of AIA images on a server, and it was occupying all the cores and making the server very slow.
I've been told that the red part is how much time is spent in the kernel rather than in the Python code itself. Could it be too many parallel jobs? Is there a way to run the code more efficiently so that it doesn't slow down the server?

[alasdairwilson]

I assume you mean you are running parallel copies of the dem code, as in you have more than one set of AIA images and you are dispatching multiple jobs to make DEM maps from each set?

If that is not correct then let me know but assuming that is correct: demregpy is already parallelised, if you have enough pixels for it to be worth it then your data arrays are chunked into 100 pixel blocks and these are sent to cpu cores. It attempts to utilise all the system cores in this manner.

Since the occupancy is already very high when you add a bunch more jobs, they also try to dispatch these blocks to the threads that are already busy and this leads to far far slower behaviour than one job. This would manifest as a bunch of time spent on system tasks (which is seen here). This would be the same if you tried to parallelise a bunch of scipy or numpy matrix math which are already parallelised.

Since you have 64 cores and I chunk it in 100 pixel chunks then 64,000 pixels will be processed at any given time. If your images contain significantly more than this (e.g. 2-4x as many pixels as that) then you are probably already very efficient with more pixels being more efficient up to a limit. If they are smaller than that not then you might want to run multiple jobs at the same time.

To do so, you need to limit the number of threads available to demregpy manually via threadpoolctl. e.g. if you want to run 4 at a time then you should limit it to 16 threads (64/4 = 16). Otherwise, just leave it be with a serial list of observations to do one at a time.

[MohamedNedal]

Hi @alasdairwilson I was running one script only to do DEM analysis for around 600 frames, 12-second cadence AIA images in 6 channels within a 2-hour duration. Is there a way to make the code use fewer CPU cores fully rather than using all the cores partially?
For example, are any of these two ways (or both together) correct?
1.

import os

os.environ['OMP_NUM_THREADS'] = '16'

from threadpoolctl import threadpool_limits

with threadpool_limits(limits=16):
    # DEM analysis code here
    ...
    pass

[nawinnova]

Hi Mohammed and Alasdair

Not sure if this is still need, but I found similar issues and find a workaround on it.

In my case it seems like the numpy is actually multi-threading inside the processpoolexecutor function in demmap_pos.py, the multithread nature of numpy make it seems to occupy every CPU core possible even if you set max_workers argument in processpoolexecutor to be much less than actual CPU.

The workaround that I found is to specify a function as follows in demmap_pos.py, just after import function lines

def init_worker():
 threadpool_limits(limits=1)

and then at line 117

with ProcessPoolExecutor(max_workers=80, initializer=init_worker) as exe:
This will ensure that each child process generate by ProcessPoolExecutor has threadpool_limits(limits=1). Note that I also identify number of max_workers to ensure that it belows number of core by considerable amonunt (I ran on 128 core cpu)

I have tried this method, and I found that the CPU time increase to 90%, as well as my DEM computation speed become much faster(~30000 DEM/s ) and there are still some CPUs leftover for other users.

Note that I have not test extensively, but this could be something that you can experiment with.

Hope it helps!

Best wishes,
Nawin

[ianan]

I've added a max_workers option as per @nawinnova suggestion above - check this branch, testing it here https://github.com/ianan/demreg/blob/29-code-in-parallel/python/test_syn_parallel.ipynb

Set it up to use for default 50% of the CPU/threads the machine has when running in parallel, or you can specify how many you want via max_workers. Let me know if this works, or needing tweaking. Need to double check this on more hardware.

I guess could also add check that max_works isn't bigger than multiprocessing.cpu_count() ?

If ok then can create PR and merge with the main branch.