Skip to content

Implement string tokenization functionalities as defined in Fortran 2023 - #443

Merged
kuanchihwang merged 12 commits into
ESCOMP:developmentfrom
kuanchihwang:staging/mpas-dycore-split-tokenize
Dec 18, 2025
Merged

Implement string tokenization functionalities as defined in Fortran 2023#443
kuanchihwang merged 12 commits into
ESCOMP:developmentfrom
kuanchihwang:staging/mpas-dycore-split-tokenize

Conversation

@kuanchihwang

@kuanchihwang kuanchihwang commented Nov 11, 2025

Copy link
Copy Markdown
Collaborator

Tag name (required for release branches):

None

Originator(s):

kuanchihwang

Descriptions (include the issue title, and the keyword ['closes', 'fixes', 'resolves'] followed by the issue number):

This PR implements the new string tokenization functionalities that have been standardized in Fortran 2023. Specifically, two new intrinsic procedures have been defined with the following signatures:

  1. split (string, set, pos [, back]), defined in section 16.9.196.
  2. tokenize (string, set, tokens [, separator]) or tokenize (string, set, first, last), defined in section 16.9.210.

The implementations provided by this PR should be fully compliant with the specifications in Fortran 2023.

Additionally, the parse_stream_name function, which is part of the I/O infrastructure for MPAS dynamical core, is refactored to use the tokenize subroutine for better robustness.

Describe any changes made to the build system:

None

Describe any changes made to the namelist:

None

List any changes to the defaults for the input datasets (e.g., boundary datasets):

None

List all files eliminated and why:

None

List all files added and what they do:

None

List all existing files that have been modified, and describe the changes:

M       src/core_utils/string_core_utils.F90
M       src/utils/string_utils.F90
M       test/unit/fortran/src/core_utils/test_string_core_utils.pf
  * Avoid unnecessary renames from `stringify` to `core_stringify`
  * Also make `split` and `tokenize` available in `string_utils` module
M       src/dynamics/mpas/driver/dyn_mpas_procedures.F90
  * Implement `split` and `tokenize` from Fortran 2023
M       src/dynamics/mpas/driver/dyn_mpas_subdriver.F90
  * Rewrite `parse_stream_name` in terms of `tokenize`
M       src/dynamics/mpas/dyn_procedures.F90
  * Add missing dates
M       src/dynamics/mpas/tests/unit/test_dyn_mpas_procedures.pf
  * Implement unit tests for `split` and `tokenize`

Regression tests:

  SMS_Ln9.ne3pg3_ne3pg3_mg37.FKESSLER.derecho_intel.cam-outfrq_se_cslam_multitape (Overall: NLFAIL)

Except for the known failing test above, all the other tests pass with respect to the last baseline, sima0_09_000.

@kuanchihwang
kuanchihwang marked this pull request as ready for review November 11, 2025 22:58
@kuanchihwang

Copy link
Copy Markdown
Collaborator Author

To reviewers, you can check out the rendered code documentation here for split and here for tokenize.

@jimmielin

Copy link
Copy Markdown
Collaborator

Thanks @kuanchihwang. I wonder what made the GNU regression tests different (it looks like the snapshot runs are fine which is great!). Maybe we have to accept the new baselines as it is difficult to know what changed at this point.

help@ hasn't gotten back to me yet about the libhugetlbfs.so issue I've been running to that crashes GNU runs at runtime, but it looks like the file is now available on the compute nodes again (I was able to run a case as well). Maybe they are still in the progress of fixing issues?

@nusbaume

Copy link
Copy Markdown
Collaborator

Hi @kuanchihwang @jimmielin, I just ran the SIMA regression tests with the head of the ESCOMCAM-SIMA development branch and I got the same set of failures, so my guess is that the underlying libraries changed somehow during the update and that is what is causing the differences (I agree that it is not related to the changes in this PR).

I have no idea why it is SE-CSLAM specific, but the magnitudes seem quite small (largest relative differences of ~1e-7), and given that the dycore is out-of-date anyways I am not too worried about it at this stage, as long as it is recorded somewhere (e.g. here) so that if we need to trace any differences back we can.

@jimmielin

Copy link
Copy Markdown
Collaborator

Sounds good, thanks @nusbaume and @kuanchihwang! I agree if it's just SE it's probably fine, at least we know the physics code remains bit-for-bit.

jimmielin pushed a commit that referenced this pull request Dec 1, 2025
Tag name (required for release branches): sima0_09_000
Originator(s): @jimmielin

Description (include the issue title, and the keyword ['closes',
'fixes', 'resolves'] followed by the issue number):
- Completes CCPPization of gravity wave drag parameterizations for CAM4
through CAM7.
- Ports frontogenesis bug fix for SE dycore from
ESCOMP/CAM#1415
- Adds vorticity calculation for gravity waves from existing CAM SE
dycore.

Describe any changes made to build system: N/A

Describe any changes made to the namelist: NLCOMP will fail due to added
gamma file `/gw_drag_input_nl/ bnd_rdggm`

List any changes to the defaults for the input datasets (e.g. boundary
datasets): N/A

List all files eliminated and why: N/A

List all files added and what they do: see below with `A` flag

List all existing files that have been modified, and describe the
changes:
(Helpful git command: `git diff --name-status
development...<your_branch_name>`)
```
M       cime_config/testdefs/testlist_cam.xml
A       cime_config/testdefs/testmods_dirs/cam/outfrq_gw_cam4_derecho/shell_commands
A       cime_config/testdefs/testmods_dirs/cam/outfrq_gw_cam4_derecho/user_nl_cam
A       cime_config/testdefs/testmods_dirs/cam/outfrq_gw_cam7_derecho/shell_commands
A       cime_config/testdefs/testmods_dirs/cam/outfrq_gw_cam7_derecho/user_nl_cam
  - add tests for CAM4 and CAM7 gravity wave schemes

M       cime_config/testdefs/testmods_dirs/cam/outfrq_se_cslam_analy_ic/user_nl_cam
  - add topo file to analytic ic FCAM7 test for now. this should be removed 
    in the future when FCAM7 test runs standalone without analytic ICs.

M       src/control/runtime_obj.F90
M       src/control/runtime_opts.F90
  - remove gw_front and gw_front_igw from runtime object. 
    This controls the provisioning of frontogenesis fields from the SE dycore to the physics state.
  - These two parameters were not read from namelist and were controlled 
    independently from the schemes, and changing them would not enable/disable
    the frontogenesis gw scheme, so I removed them to avoid further confusion. 
    I think this needs a more elegant solution to determine active schemes
    from the host model end. 

M       src/data/registry.xml

M       src/dynamics/se/dp_coupling.F90
M       src/dynamics/se/dyn_comp.F90
M       src/dynamics/se/gravity_waves_sources.F90
  - always provide frontgf, frontga to physics state
  - port vorticity code (vort4gw) for moving mountains gravity wave parameterization
    from CAM SE dycore to SIMA.

M       src/control/cam_comp.F90
A       src/physics/utils/gravity_wave_drag_ridge_read.F90
A       src/physics/utils/gravity_wave_drag_ridge_read.meta
M       cime_config/namelist_definition_cam.xml
  - add host-model utility to read topo and gamma ridge file (on model decomp) 
    for gravity wave parameterizations.

M       src/data/ref_pres.F90
M       src/data/ref_pres.meta
  - add bottom pressure and index of gravity wave tapering at top of model to ref_pres.
  - change standard name of do_molec_diff to do_molecular_diffusion.

M       test/unit/python/sample_files/ref_pres.meta
M       test/unit/python/sample_files/ref_pres_SourceMods.meta
  - change standard name of do_molec_diff to do_molecular_diffusion.
```

If there are new failures (compared to the
`test/existing-test-failures.txt` file),
have them OK'd by the gatekeeper, note them here, and add them to the
file.
If there are baseline differences, include the test and the reason for
the
diff. What is the nature of the change? Roundoff?

derecho/intel/aux_sima:
```
  SMS_Ln9.mpasa480_mpasa480.FKESSLER.derecho_intel.cam-outfrq_kessler_mpas_derecho (Overall: NLFAIL) details:
  SMS_Ln9.ne3pg3_ne3pg3_mg37.FADIAB.derecho_intel.cam-outfrq_se_cslam (Overall: NLFAIL) details:
  SMS_Ln9.ne3pg3_ne3pg3_mg37.FHS94.derecho_intel.cam-outfrq_se_cslam (Overall: NLFAIL) details:
  SMS_Ln9.ne3pg3_ne3pg3_mg37.FKESSLER.derecho_intel.cam-outfrq_se_cslam (Overall: NLFAIL) details:
  SMS_Ln9.ne3pg3_ne3pg3_mg37.FTJ16.derecho_intel.cam-outfrq_se_cslam (Overall: NLFAIL) details:
  - new namelist group gw_drag_input_nl

  SMS_Ln9.ne3pg3_ne3pg3_mg37.FCAM4.derecho_intel.cam-outfrq_se_cslam_analy_ic_cam4 (Overall: DIFF) details:
  SMS_Ln9.ne3pg3_ne3pg3_mg37.FCAM7.derecho_intel.cam-outfrq_se_cslam_analy_ic (Overall: DIFF) details:
  - answer change: SDFs added gravity wave drag parameterizations


  SMS_Ln9.ne3pg3_ne3pg3_mg37.FKESSLER.derecho_intel.cam-outfrq_se_cslam_multitape (Overall: NLFAIL) details:
  - pre-existing failure due to NLCOMP not supporting multiple history tapes
```

derecho/gnu/aux_sima:
```
  SMS_Ln2.ne3pg3_ne3pg3_mg37.FPHYStest.derecho_gnu.cam-outfrq_dme_adjust_derecho (Overall: NLFAIL) details:
  SMS_Ln2.ne3pg3_ne3pg3_mg37.FPHYStest.derecho_gnu.cam-outfrq_hack_shallow_derecho (Overall: NLFAIL) details:
  SMS_Ln2.ne3pg3_ne3pg3_mg37.FPHYStest.derecho_gnu.cam-outfrq_hb_vdiff_derecho (Overall: NLFAIL) details:
  SMS_Ln2.ne3pg3_ne3pg3_mg37.FPHYStest.derecho_gnu.cam-outfrq_held_suarez_derecho (Overall: NLFAIL) details:
  SMS_Ln2.ne3pg3_ne3pg3_mg37.FPHYStest.derecho_gnu.cam-outfrq_kessler_derecho (Overall: NLFAIL) details:
  SMS_Ln2.ne3pg3_ne3pg3_mg37.FPHYStest.derecho_gnu.cam-outfrq_rk_stratiform_derecho (Overall: NLFAIL) details:
  SMS_Ln2.ne3pg3_ne3pg3_mg37.FPHYStest.derecho_gnu.cam-outfrq_tj2016_after_coupler_derecho (Overall: NLFAIL) details:
  SMS_Ln2.ne3pg3_ne3pg3_mg37.FPHYStest.derecho_gnu.cam-outfrq_tj2016_before_coupler_derecho (Overall: NLFAIL) details:
  SMS_Ln2.ne3pg3_ne3pg3_mg37.FPHYStest.derecho_gnu.cam-outfrq_zm_derecho (Overall: NLFAIL) details:
  SMS_Ln9.mpasa480_mpasa480.FKESSLER.derecho_gnu.cam-outfrq_kessler_mpas_derecho (Overall: NLFAIL) details:
  - new namelist group gw_drag_input_nl

  SMS_Ln2.ne3pg3_ne3pg3_mg37.FPHYStest.derecho_gnu.cam-outfrq_gw_cam4_derecho (Overall: DIFF) details:
  SMS_Ln2.ne3pg3_ne3pg3_mg37.FPHYStest.derecho_gnu.cam-outfrq_gw_cam7_derecho (Overall: DIFF) details:
  - new baselines for new physics test (gravity waves for CAM4 and CAM7 SE).

  SMS_Ln9.ne3pg3_ne3pg3_mg37.FCAM4.derecho_gnu.cam-outfrq_se_cslam_analy_ic_cam4 (Overall: DIFF) details:
  SMS_Ln9.ne3pg3_ne3pg3_mg37.FCAM7.derecho_gnu.cam-outfrq_se_cslam_analy_ic (Overall: DIFF) details:
  - answer change: SDFs added gravity wave drag parameterizations

  SMS_Ln9.ne3pg3_ne3pg3_mg37.FADIAB.derecho_gnu.cam-outfrq_se_cslam (Overall: DIFF) details:
  SMS_Ln9.ne3pg3_ne3pg3_mg37.FHS94.derecho_gnu.cam-outfrq_se_cslam (Overall: DIFF) details:
  SMS_Ln9.ne3pg3_ne3pg3_mg37.FKESSLER.derecho_gnu.cam-outfrq_se_cslam (Overall: DIFF) details:
  SMS_Ln9.ne3pg3_ne3pg3_mg37.FTJ16.derecho_gnu.cam-outfrq_se_cslam (Overall: DIFF) details:
  - SE dycore answer differences after Derecho upgrade, unrelated to this PR.
  - new baselines are being established. see #443 (comment)
```

If this changes climate describe any run(s) done to evaluate the new
climate in enough detail that it(they) could be reproduced:

CAM-SIMA date used for the baseline comparison tests if different than
latest:

---------

Co-authored-by: Jesse Nusbaumer <nusbaume@ucar.edu>
@jimmielin

Copy link
Copy Markdown
Collaborator

Hi @kuanchihwang I pushed the changed GNU baselines post-Derecho upgrade as part of sima0_09_000 (#426) so hopefully you will have all GNU tests bit-for-bit now. Thanks!

@nusbaume nusbaume left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @kuanchihwang! I just had a few, hopefully simple, requests.

Comment thread src/core_utils/string_core_utils.F90 Outdated
Comment thread src/core_utils/string_core_utils.F90
Comment thread src/dynamics/mpas/driver/dyn_mpas_procedures.F90
Comment thread src/dynamics/mpas/tests/unit/test_dyn_mpas_procedures.pf
Comment thread src/dynamics/mpas/tests/unit/test_dyn_mpas_procedures.pf Outdated

@nusbaume nusbaume left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the extra comments and tests @kuanchihwang! I had one last (optional) test string modification request to try and maximize test coverage, but otherwise this PR looks good to me.

Comment thread src/dynamics/mpas/tests/unit/test_dyn_mpas_procedures.pf Outdated
@kuanchihwang

Copy link
Copy Markdown
Collaborator Author

@nusbaume It seems that I need one more approving review to proceed with the PR. Would you like to suggest an additional reviewer? Thanks!

@nusbaume

Copy link
Copy Markdown
Collaborator

@kuanchihwang just letting you know that your PR should be ready to go now. Thanks for bringing it in!

@kuanchihwang
kuanchihwang merged commit 31d38b5 into ESCOMP:development Dec 18, 2025
18 checks passed
@kuanchihwang
kuanchihwang deleted the staging/mpas-dycore-split-tokenize branch December 18, 2025 21:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants