sima0_13_003: Fix model crash in MPAS dynamical core and implement assorted code cleanup by kuanchihwang · Pull Request #486 · ESCOMP/CAM-SIMA

kuanchihwang · 2026-03-31T20:31:53Z

Tag name (required for release branches):

sima0_13_003 due to newly-added tests, otherwise bit-for-bit.

Originator(s):

kuanchihwang

Descriptions (include the issue title, and the keyword ['closes', 'fixes', 'resolves'] followed by the issue number):

This PR fixes a model crash in MPAS dynamical core, which is caused by a regression in recent versions of GNU Fortran compiler. When CAM-SIMA is configured with MPAS dynamical core, and is built with GNU Fortran compiler version 14 in debug mode, running the model will lead to a segmentation fault. However, the crash mysteriously disappears when the model is built in optimized mode. Older GNU Fortran compiler version 12 does not exhibit this behavior, neither does Intel Fortran compiler.

Closes #484

Additionally, this PR also brings the following:

Update Python script for generating MPAS namelist definition file
Reduce API surface area by making dyn_inquire_mesh_dimensions private
Add lookup table for mapping indexes from advected to all constituents and refactor the rest of code to use it
Fix incorrect constituent indexing inside set_physics_state_external during dynamics-physics coupling

The rationale behind each change is described in the commit messages.

Describe any changes made to the build system:

None

Describe any changes made to the namelist:

None

List any changes to the defaults for the input datasets (e.g., boundary datasets):

None

List all files eliminated and why:

None

List all files added and what they do:

None

List all existing files that have been modified, and describe the changes:

M       src/dynamics/mpas/assets/generate_namelist_definition.py
  * Update Python script for generating MPAS namelist definition file
M       src/dynamics/mpas/dyn_comp.F90
  * Add lookup table for mapping indexes from advected to all constituents
M       src/dynamics/mpas/dyn_comp_impl.F90
  * Add lookup table for mapping indexes from advected to all constituents
  * Fix MPAS dycore crash due to regression in recent GNU Fortran compiler
M       src/dynamics/mpas/dyn_coupling_impl.F90
  * Fix indentation
  * Add lookup table for mapping indexes from advected to all constituents
  * Fix MPAS dycore crash due to regression in recent GNU Fortran compiler
  * Fix incorrect constituent indexing
M       src/dynamics/mpas/dyn_grid.F90
  * Make `dyn_inquire_mesh_dimensions` private
M       src/dynamics/mpas/dyn_grid_impl.F90
  * Make `dyn_inquire_mesh_dimensions` private
M       src/dynamics/mpas/dyn_procedures.F90
  * Fix MPAS dycore crash due to regression in recent GNU Fortran compiler
M       src/dynamics/mpas/tests/unit/test_dyn_procedures.pf
  * Fix MPAS dycore crash due to regression in recent GNU Fortran compiler

Regression tests:

SMS_Ln9.ne3pg3_ne3pg3_mg37.FADIAB.derecho_gnu.cam-outfrq_se_cslam (Overall: FAIL)
SMS_Ln9.ne3pg3_ne3pg3_mg37.FKESSLER.derecho_intel.cam-outfrq_se_cslam_multitape (Overall: NLFAIL)

Known failing tests.

SMS_D_Ln9.mpasa120_mpasa120.QPC4.derecho_gnu.cam-outfrq_analy_ic_cam4 (Overall: DIFF)
SMS_D_Ln9.mpasa120_mpasa120.QPC4.derecho_intel.cam-outfrq_analy_ic_cam4 (Overall: DIFF)

Newly-added tests.

The `entry_id_pg` element in the namelist definition XML file was updated to version 2.0 in PR 460. However, the corresponding Python script was not updated. After updating the Python script, it is also confirmed that: 1. The namelist definition XML file is up to date. 2. The namelist definition XML file is successfully validated against the XML schema at CIME/ParamGen/xml_schema/entry_id_pg.xsd.

The `dyn_inquire_mesh_dimensions` procedure does not need to be a public API because nothing outside the `dyn_grid` module uses it. Privatize it to reduce API surface area.

CAM-SIMA holds information about all constituents. However, MPAS dynamical core only knows about the advected ones. This lookup table maps an index within the set of advected constituents to an index within the set of all constituents. Refactor the rest of code to use it, removing duplicate and error-prone code logic throughout.

When CAM-SIMA is configured with MPAS dynamical core, and is built with GNU Fortran compiler version 14 in debug mode, running the model will lead to a segmentation fault. However, the crash mysteriously disappears when the model is built in optimized mode. Older GNU Fortran compiler version 12 does not exhibit this behavior, neither does Intel Fortran compiler. This issue is caused by a regression in recent GNU Fortran compiler, where a procedure call involving an array argument with indirect indexing is mishandled. Split out the operations into simpler steps to fix this issue.

The `mpas_dynamical_core % map_constituent_index` type-bound function maps MPAS scalar index to CAM-SIMA *advected* constituent index, but we are still one step away. Should also use the `advected_constituent_index` lookup table to map *advected* constituent index to *all* constituent index.

nusbaume

Thanks for the bug fix and cleanup @kuanchihwang! To help us catch these potential issues in the future, I was wondering if we could add another set of MPAS+CAM4 regression tests, but this time with debug flags enabled? This should be doable by just adding the following lines to CAM-SIMA/cime_config/testdefs/testlist_cam.xml, ideally at the end of the CAM4 physics tests section:

<test compset="QPC4" grid="mpasa120_mpasa120" name="SMS_D_Ln9" testmods="cam/outfrq_analy_ic_cam4">
    <machines>
      <machine name="derecho" compiler="intel" category="aux_sima"/>
      <machine name="derecho" compiler="gnu" category="aux_sima"/>
    </machines>
    <options>
      <option name="wallclock">00:20:00</option>
      <option name="comment">Aquaplanet CAM4 test with MPAS dycore at 120 km resolution  with compiler debug flags enabled</option>
    </options>
 </test>

I also had one code-related question, but otherwise everything else looked good to me. Thanks again!

nusbaume

Everything looks great to me now. Thanks again @kuanchihwang!

jimmielin

Thanks @kuanchihwang!

kuanchihwang-ncar added 6 commits March 31, 2026 14:13

Fix indentation

5c231b9

Make dyn_inquire_mesh_dimensions private

dff8b3e

The `dyn_inquire_mesh_dimensions` procedure does not need to be a public API because nothing outside the `dyn_grid` module uses it. Privatize it to reduce API surface area.

kuanchihwang temporarily deployed to CI-tests-on-CIRRUS March 31, 2026 20:32 — with GitHub Actions Inactive

kuanchihwang marked this pull request as ready for review March 31, 2026 20:40

kuanchihwang requested a review from nusbaume March 31, 2026 20:40

nusbaume requested changes Apr 10, 2026

View reviewed changes

Comment thread src/dynamics/mpas/dyn_comp_impl.F90 Outdated

kuanchihwang-ncar added 2 commits April 12, 2026 23:39

Remove unnecessary array

276177d

Add CIME test for CAM4 aquaplanet with MPAS dynamical core in debug mode

bc4bef9

kuanchihwang temporarily deployed to CI-tests-on-CIRRUS April 13, 2026 06:41 — with GitHub Actions Inactive

jimmielin self-requested a review April 13, 2026 16:56

nusbaume self-requested a review April 13, 2026 16:57

nusbaume approved these changes Apr 13, 2026

View reviewed changes

jimmielin approved these changes Apr 13, 2026

View reviewed changes

kuanchihwang changed the title ~~Fix model crash in MPAS dynamical core and implement assorted code cleanup~~ sima0_13_003: Fix model crash in MPAS dynamical core and implement assorted code cleanup Apr 14, 2026

kuanchihwang merged commit e1fa97d into ESCOMP:development Apr 14, 2026
19 checks passed

kuanchihwang deleted the staging/mpas-dycore-fix-indirect-indexing-crash branch April 15, 2026 00:20

Navigation Menu

Uh oh!

sima0_13_003: Fix model crash in MPAS dynamical core and implement assorted code cleanup - #486

sima0_13_003: Fix model crash in MPAS dynamical core and implement assorted code cleanup#486
kuanchihwang merged 8 commits into
ESCOMP:developmentfrom
kuanchihwang:staging/mpas-dycore-fix-indirect-indexing-crash

kuanchihwang commented Mar 31, 2026 •

edited

Loading

Uh oh!

nusbaume left a comment

Uh oh!

Uh oh!

nusbaume left a comment

Uh oh!

jimmielin left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

kuanchihwang commented Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Tag name (required for release branches):

Originator(s):

Descriptions (include the issue title, and the keyword ['closes', 'fixes', 'resolves'] followed by the issue number):

Describe any changes made to the build system:

Describe any changes made to the namelist:

List any changes to the defaults for the input datasets (e.g., boundary datasets):

List all files eliminated and why:

List all files added and what they do:

List all existing files that have been modified, and describe the changes:

Regression tests:

Uh oh!

nusbaume left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

nusbaume left a comment

Choose a reason for hiding this comment

Uh oh!

jimmielin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kuanchihwang commented Mar 31, 2026 •

edited

Loading