Skip to content

MAGE coupling #695

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
fvitt wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

MAGE coupling #695

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c9f5217
Split MPI communicator for MAGE coupling
fvitt Aug 15, 2026
a7c0b51
Merge remote-tracking branch 'ESCOMP/main' into waccmx-mage
fvitt Aug 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions cesm/driver/esmApp.F90
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,8 @@ program esmApp
#else
call MPI_init(rc)
#endif
! note: this is superseded by the MPI_COMM_split below, which restricts this
! application to its own subset of MPI_COMM_WORLD
COMP_COMM = MPI_COMM_WORLD
Comment on lines +44 to 46

@billsacks billsacks Aug 18, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any value in maintaining this line given that it is superseded below?


!-----------------------------------------------------------------------------
Expand All @@ -49,6 +51,27 @@ program esmApp

! by default, ESMF_LOGKIND_MULTI_ON_ERROR does not create files PET[N*].ESMF_LogFile unless there is an error
! if want those files, comment out the following line and uncomment the line logkindflag = ESMF_LOGKIND_MULTI

!-----------------------------------------------------------------------------
! Restrict this application to its own subset of MPI_COMM_WORLD.
!
! WACCM-X/MAGE runs as a multiple-program (MPMD) job -- the CESM executable and
! the MAGE executables (Gamera, REMIX, RCM) are launched together, so
! MPI_COMM_WORLD spans all of the models. Splitting on the WACCM-X application
! ID hands CESM a communicator containing only its own ranks; that is the
! communicator passed to ESMF_Initialize below, so the entire CESM component
! tree runs on it. MPI_COMM_WORLD itself is deliberately left untouched
! because the cross-model coupling in CAM (src/ionosphere/waccmx/mage_module.F90)
! uses it directly as the pool from which the WACCM-X <-> REMIX coupling
! communicator is built.
!
! The color (67) must match myAppId in mage_module.F90; the other models split
! the same world with their own IDs. A key of 0 preserves the existing rank
! ordering. In a standalone CESM run every rank passes the same color, so the
! result is simply a duplicate of MPI_COMM_WORLD and behavior is unchanged.
!-----------------------------------------------------------------------------
call MPI_COMM_split(MPI_COMM_WORLD,67,0,COMP_COMM,ier)

@billsacks billsacks Aug 18, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to the documentation of this MPI routine, it looks like this does an allgather. Is that a performance concern? If we keep this method, I'm wondering if we should put it in a conditional on some flag that is only enabled when it's actually needed to avoid this unnecessary complexity and allgather. Something like:

if (running_mpmd) then
   call MPI_COMM_split(MPI_COMM_WORLD,67,0,COMP_COMM,ier)
else
   COMP_COMM = MPI_COMM_WORLD
end if

(Note that that block would also replace the above setting of COMP_COMM = MPI_COMM_WORLD a few lines above... it assumes that we don't need an initial setting of COMP_COMM = MPI_COMM_WORLD if we're about to call MPI_COMM_split; I don't know if that's the case or not.)

(The name of the logical – running_mpmd – may not be the best / most appropriate.)

I'm also wondering if, for this relatively simple situation, there might be a better routine to use than MPI_COMM_split, which seems overly general for this scenario. Can this be done via MPI_Comm_create? I ask this totally naively, since I don't really understand this.


call mpi_comm_rank(COMP_COMM, iam, ier)
if (iam==0) then
open(newunit=fileunit, status="old", file="drv_in")
Expand Down
Loading