MAGE coupling

[fvitt]

Description of changes

Split MPI communicator to allow 2-way coupling of WACCM-X with MAGE magnetosphere in a MPMD job.

Specific notes

See comments added to cesm/driver/esmApp.F90

Contributors other than yourself, if any: @phamkh

CMEPS Issues Fixed (include github issue #):

Are changes expected to change answers? (specify if bfb, different at roundoff, more substantial)
bit-for-bit

Any User Interface Changes (namelist or namelist defaults changes)?
N/A

Testing performed

aux_cam on izumi and derecho

[billsacks]

Based on input from @cacraigucar , this is connected with ESCOMP/CAM#1649. The initial review of that CAM PR has questioned whether we actually want to take the current approach. I'm going to mark this as a draft until this decision is resolved.

In the meantime, I had already reviewed this code and had some questions / thoughts. I'm sharing these, but these don't need any action unless / until we decide to move ahead with this.

[billsacks]

Is there any value in maintaining this line given that it is superseded below?

[billsacks]

According to the documentation of this MPI routine, it looks like this does an allgather. Is that a performance concern? If we keep this method, I'm wondering if we should put it in a conditional on some flag that is only enabled when it's actually needed to avoid this unnecessary complexity and allgather. Something like:

if (running_mpmd) then
   call MPI_COMM_split(MPI_COMM_WORLD,67,0,COMP_COMM,ier)
else
   COMP_COMM = MPI_COMM_WORLD
end if

(Note that that block would also replace the above setting of COMP_COMM = MPI_COMM_WORLD a few lines above... it assumes that we don't need an initial setting of COMP_COMM = MPI_COMM_WORLD if we're about to call MPI_COMM_split; I don't know if that's the case or not.)

(The name of the logical – running_mpmd – may not be the best / most appropriate.)

I'm also wondering if, for this relatively simple situation, there might be a better routine to use than MPI_COMM_split, which seems overly general for this scenario. Can this be done via MPI_Comm_create? I ask this totally naively, since I don't really understand this.