Particle Push: OpenMP Not Working if no Collective Effects

[ax3l]

Tested with 1 MPI rank:
It looks like setting OMP_NUM_THREADS to values greater than 1 neither accelerates tracking nor space charge calculations right now. This is a defect.

The init logic shows we use multiple threads, but tracking takes as long with 1 as with 4 threads as of 25.02.

Reproducer:

• any test w/o collective effects on CPU, e.g., FODO Cell

Reproducer (private link):

• https://github.com/ECP-WarpX/impactx-benchmarks/blob/main/spacecharge_impactx.py
• https://github.com/ECP-WarpX/impactx-benchmarks/blob/main/drift_impactx.py
  But any other example (disable diagnostics to reduce noise) shows this as well.

[ax3l]

Update: @atmyers is working on this right now.

For runs w/o collective effects, we do never call Redistribute to keep a perfect load balance of particles.

The problem stems from particle init, where we have to assign particles to OMP_NUM_THREAD tiles to make sure we have tiles available for OpenMP threads.

The init logic is here:
https://github.com/ECP-WarpX/impactx/blob/25.02/src/initialization/InitDistribution.cpp#L281-L314
And identically here:
https://github.com/ECP-WarpX/impactx/blob/25.02/src/elements/Source.cpp#L116-L120

Calling the same AddNParticles using device vectors:
https://github.com/ECP-WarpX/impactx/blob/25.02/src/particles/ImpactXParticleContainer.cpp#L122-L202