merge to ESCOMP cmeps1.1.41 - #46
Merged
mvertens merged 69 commits intoApr 9, 2026
Merged
Conversation
separation of mct and nuopc data, xcpl and stub compoents
…#3547) Update CIME to ESMCI cime5.8.20 Squash merge of jgfouca/branch-for-to-acme-2020-04-14 This is the first cime update in half a year, lots of changes. Features: * allow for specification of vertical grid in ATM_GRID * if no env vars, print comment to assist user in env_mach_specific.xml * case.setup: add --keep option for clean/reset actions to keep local changes to generated files * case.setup: refresh build Macros when doing a reset/clean * cprnc: cime will build a copy in sharelibs if no global version available * check_input_data: add ftp timeout, make sure failover to svn is working * added support for new OMIP compsets in POP as well as better handling of the different versions of JRA forcings * Pio2 updated to latest master * reorganize mct and nuopc data, xcpl and stub components based on coupler * new auto-triaging feature for tests, will report potentially broken commits to TestStatus.log * coupler: Allow writing T-forcing cplhist files even when running with SGLC * Add support for UFS model! Not really relevant to E3SM, but is a major development in CIME * archiver: Extend archiving support to accept a regex pattern * coupler: Update for compatibility with latest CMEPS Bug fixes: * check_input_data: fix issue with pop input files that should not be found * fix default for PIO_REARRANGER * docn: Small fix to allow scam to run with aquaplanet * fix issue with resubmit on systems that use ssh * fix misleading output in CaseStatus * Update description for FORCE_BUILD_SMP * archiver: pop dd history files were not identified by get_extension * NOTE: MPAS history files are now being processed as they should have been all along * Improve scripts_regression_tests test cleanup * dlnd: Some gnu tests were giving an error of different character lengths in dlnd. * coupler: seq flux should be initialized even if for some reason the namelist isn't read * env_batch.xml: we can now handle the case where multiple queues have the same name * ERI: fix tests that were less than a day, or with START_TOD nonzero * test hist: Exclude .ref1 and .ref2 from TestStatus files * coupler: fix issue with e3sm field Fixes: #2324 [BFB]
Update fork to latest cime for PR to fix FV3 log error in cime.
FV3 bug fix and Makefile update One line change to C96_C96_mg17 grid alias and a mod to cime/scripts/Tools/Makefile to get rid of an erroneous warning when building a configuration without an active atmosphere model. This PR closes #3452. Test suite: Standart CIME regression tests and foo test listed by Jim in the original issue. Also ran an exact restart test using the fixed C96 alias. Test baseline: Master Test namelist changes: na Test status: [bit for bit, roundoff, climate changing] OK all passed. Ran 156 tests in 3008.707s OK (skipped=14) Fixes [CIME Github issue #3452 ] User interface changes?: na Update gh-pages html (Y/N)?:N Code review:
* esmci_remote_for_split/master: (2458 commits) fix cime_model to stdout issue extend allowed string length Add a comment in the Makefile improve format fix formatting revert path change add make check, fix tests fix format issues change type to double for derived vars Updated filter on calculating normalized rms. Now, a variable must be >0, not nearly 0, to be normalized. Also, variables that cannot be normalized generate a huge rms. Added check that the normalizing denominator is >0 when calculating a relative rms Fix typo Update for cime5.8.23 Missing credit in shr_flux_atmOcn fix more pylint issues, unrelated to pr fix pylint issue fix atm level spec so that it does not match 1x1_brazil Add missing ROF2OCN grid for r05_TO_gx3v7 fix for issue 3452 erroneous log error due to FV3 additions to Makefile, and bug fix for FV3 compset definition. Update NCPL values ...
Nuopc timing cheyenne esmf
changes for nuopc xcpl_comps needed for new ice sheets capability
nag_port for nuopc Fix issue with github actions, add support for ESMF on izumi. Test suite: scripts_regression_tests.py with CIME_DRIVER=nuopc and intel,gnu,nag compilers and mpi-serial and mvapich2 I could not get the pgi compiler, the nag 7.0 compiler or the openmpi libraries working. Test baseline: Test namelist changes: Test status: bit for bit Fixes User interface changes?: Update gh-pages html (Y/N)?: Code review:
Except the 'tools' subdirectory. Not sure that code is ready for py3
Adds enforcing code formatting to pre-commit The goal of this PR is to enforce a coding style using pre-commit plugins (check-xml, end-of-file-fixed, trailing-whitespace and black). Fixing linting/formatting issues is now a single command pre-commit run -a. This auto formatting is performed by Black which enforces a strict subset of PEP8. Pre-commit runs all of these plugins for every PR. Pylint still runs as a redundancy. By enforcing a uniformed coding style PR's will now be style agnostic and less development time used on formatting code. Running pre-commit pip install pre-commit pre-commit run -a References https://black.readthedocs.io/en/stable/index.html https://black.readthedocs.io/en/stable/the_black_code_style/current_style.html Test suite: n/a Test baseline: n/a Test namelist changes: n/a Test status: n/a Fixes n/a User interface changes?: n/a Update gh-pages html (Y/N)?: N
The goal of this PR is to consolidate all files required to run CIME
under a single top-level directory. This updates CIME's structure to
be more like a standardized python library.
New
CIME/scripts
CIME/data
Captures configuration/templates
CIME/non_py
Contains non python code
CIME/non_py/externals/genf90 is now a submodule
tools/README.md
Moved
config moved to CIME/data/config
scripts/lib/CIME moved to CIME
scripts/Tools moved to CIME/Tools
src/build_scripts moved to CIME/build_scripts
src/CMake moved to CIME/non_py/src/CMake
src/components moved to CIME/non_py/src/components
src/share/timings moved to CIME/non_py/src/timings
src/eternals moved to CIME/non_py/externals
tools/configure moved to CIME/scripts/configure
tools/cprnc moved to CIME/non_py/cprnc
scripts/lib/*.template moved to CIME/data/templates
scripts/lib/get_tests.py moved to CIME/get_tests.py
scripts/lib/jenkins_generic_job.py moved to CIME/jenkins_generic_job.py
Removed
scripts/lib/six.py
scripts/lib/six_additions.py
Test suite: scripts_regression_tests.py
Test baseline: n/a
Test namelist changes: n/a
Test status: n/a
Fixes #4163
User interface changes?: n/a
Update gh-pages html (Y/N)?: n
Changes were mostly made by Claude Code, reviewed by myself.
This will be needed for qice_elev water tracers, which will have two ungridded dimensions (one for elevation class and one for water tracers). Code was written by Claude Code, reviewed by myself.
Precompute the vertical interpolation coefficients rather than recomputing them for each field. The main motivation is to avoid duplication of this logic when we add water tracer fields. Code mostly written by Claude Code with significant guidance from me, reviewed by myself.
This cast to real shouldn't be done since we are assigning an r8 to an r8.
The point of this is to make some minimal changes needed to get bit-for-bit results with an upcoming refactor, in which the elevation factors will be computed separately.
Roundoff-level mods of prep_glc_map_lnd2glc ### Description of changes Two roundoff-level modifications of prep_glc_map_lnd2glc to pave the way for a larger set of changes. I wanted to demonstrate that the larger set of changes is bit-for-bit, so here I have implemented the minimal set of answer-changing mods needed to get bit-for-bit results when running tests on my upcoming PR. ### Specific notes Contributors other than yourself, if any: CMEPS Issues Fixed (include github issue #): Are changes expected to change answers? (specify if bfb, different at roundoff, more substantial) - Single-precision-roundoff-level changes in configurations with CISM or DGLC (though diffs only show up in a small number of tests with DGLC) Any User Interface Changes (namelist or namelist defaults changes)? No ### Testing performed In the context of cesm3_0_alpha08e, ran aux_glc plus these additional tests: ``` SMS_Ly2.f09_g17_gris20.T1850Gg.derecho_intel SMS_Ld5.f10_f10_ais8gris4_mg37.I1850Clm50SpGag.derecho_intel.cism-test_coupling SMS_Lm13.f10_f10_mg37.I1850Clm50SpG.derecho_intel ERS_Ld5.ne30pg3_t232.B1850C_LTso.derecho_intel.allactive-decstart ERS_Ld5.ne30pg3_t232.BHISTC_LTso.derecho_intel.allactive-decstart ``` Tests passed (other than `FAIL NCK_Ly3.f09_g17_gris20.T1850Gg.derecho_gnu COMPARE_base_multiinst` which is documented as failing in the CISM ChangeLog). I didn't perform baseline comparisons for most tests, but I expect baseline comparisons to fail for most / all of these tests. I did look at baseline comparisons for `ERS_Ly7.f09_g17_gris4.T1850Gg.derecho_intel` and `SMS_D_Ly1.f09_g17_ais8.T1850Ga.derecho_gnu`. Looking at the cpl hist files, these had just single-precision-roundoff-level diffs, as expected. For `ERS_Ly7.f09_g17_gris4.T1850Gg.derecho_intel`, for example, these were: ``` RMS glc1Imp_Fgrg_rofi 7.8623E-16 NORMALIZED 6.3141E-09 RMS glc1Imp_Fgrg_rofl 9.8952E-18 NORMALIZED 1.0419E-07 RMS glc1Imp_Sg_topo 3.0520E-09 NORMALIZED 3.8516E-12 RMS glc1Exp_Flgl_qice 8.8304E-14 NORMALIZED 1.3056E-08 RMS glc1Exp_Sl_tsrf 4.2529E-06 NORMALIZED 2.5071E-08 ``` Based on testing on my upcoming branch - which effectively includes these changes (along with many other changes) - differences only show up in: - Tests with active CISM - Tests with DGLC that cross a year boundary (`ERS_Ld5.ne30pg3_t232.B1850C_LTso.derecho_intel.allactive-decstart` and `ERS_Ld5.ne30pg3_t232.BHISTC_LTso.derecho_intel.allactive-decstart`)
Using strategy 'ours' to ignore changes coming from main (which are from ESCOMP#633) because they are already incorporated into this branch.
This makes this averaging consistent with what's done in med_methods_FB_average, so we can switch to the latter without changing answers.
Convert count to r8 rather than r4 in prep_glc_avg ### Description of changes Convert count to r8 rather than r4 in prep_glc_avg. This makes this averaging consistent with what's done in med_methods_FB_average, so we can switch to the latter (in an upcoming PR) without changing answers. ### Specific notes Contributors other than yourself, if any: CMEPS Issues Fixed (include github issue #): Are changes expected to change answers? (specify if bfb, different at roundoff, more substantial) - probably bfb but has a slight potential to change answers in cases with lnd2glc coupling. Since integers up to about 16 million can be represented exactly in r4, I don't expect answer changes, but it's possible that there will be answer changes from this with some compilers. Any User Interface Changes (namelist or namelist defaults changes)? ### Testing performed Just tested the build (`SMS_Ld2_D_P8x1.f10_g37.X.green_gnu`).
…ridded_dims Generalize parts of post_rof_mod to handle fields with ungridded dims ### Description of changes This will be needed for the water tracer work. I haven't yet generalized the code for removing negative runoff; that will be done later. (This is similar to the changes in ESCOMP#628 but now for post_rof_mod.) ### Specific notes Contributors other than yourself, if any: CMEPS Issues Fixed (include github issue #): Are changes expected to change answers? (specify if bfb, different at roundoff, more substantial) - no, bfb Any User Interface Changes (namelist or namelist defaults changes)? No ### Testing performed Please describe the tests along with the target model and machine(s) If possible, please also added hashes that were used in the testing In the context of cesm3_0_alpha08e, ran: ``` ERS_Ly7.f09_g17_gris4.T1850Gg.derecho_intel SMS_Lm13.f10_f10_mg37.I1850Clm50SpG.derecho_intel MULTINOAIS_Ly2.f10_f10_ais8gris4_mg37.I1850Clm50SpRsGag.derecho_intel.cism-change_params SMS_D_Ld1.ne30pg3_t232.I1850Clm50BgcSpinup.derecho_intel.clm-cplhist SMS_Ld40.TL319_t232.C_JRA.derecho_intel ERS_D_Ld3.ne30pg3_t232.B1850C_LTso.derecho_intel.allactive-defaultio ERI.ne30pg3_t232.B1850C_LTso.derecho_intel.allactive-defaultio ERC_D_Ln9.ne30pg3_ne30pg3_mt232.FHISTC_LTso.derecho_intel.cam-outfrq9s ERI.TL319_t232.G_JRA.derecho_intel.cice-default SMS_D.TL319_t232.G_JRA_RYF.derecho_intel ``` All tests passed and were bit-for-bit, though with these failures that also failed in the baselines: ``` FAIL ERI.ne30pg3_t232.B1850C_LTso.derecho_intel.allactive-defaultio COMPARE_branch_hybrid FAIL ERI.ne30pg3_t232.B1850C_LTso.derecho_intel.allactive-defaultio COMPARE_rest_branch FAIL ERS_D_Ld3.ne30pg3_t232.B1850C_LTso.derecho_intel.allactive-defaultio COMPARE_base_rest ```
For water tracers: Add handling of rank-3 fields and refactor med_phases_prep_glc_map_lnd2glc ### Description of changes Two sets of changes that will be needed for water tracers - and in particular, for the tracer version of Flgl_qice_elev, which will be the first field in CMEPS that has two ungridded dimensions: (1) Extend various methods to support rank-3 fields (2) Refactor med_phases_prep_glc_map_lnd2glc to pre-compute vertical interpolation weights; this will be helpful when we introduce Flgl_qice_elev_wtracers to avoid duplicating this calculation of vertical interpolation weights between the non-tracer (rank-2 on the lnd grid and rank-1 on the glc grid) and tracer (rank-3 on the lnd grid and rank-2 on the glc grid) fields. ### Specific notes Contributors other than yourself, if any: Claude Code did a lot of the writing of this code, but I have reviewed it all carefully CMEPS Issues Fixed (include github issue #): Are changes expected to change answers? (specify if bfb, different at roundoff, more substantial) bfb when using ESCOMP#633 as a baseline Any User Interface Changes (namelist or namelist defaults changes)? No ### Testing performed In the context of cesm3_0_alpha08e, ran: (1) aux_glc on derecho plus a few extra tests, with comparisons against baselines generated with ESCOMP#633 . The extra tests beyond aux_glc were: ``` SMS_Ly2.f09_g17_gris20.T1850Gg.derecho_intel SMS_Ld5.f10_f10_ais8gris4_mg37.I1850Clm50SpGag.derecho_intel.cism-test_coupling SMS_Lm13.f10_f10_mg37.I1850Clm50SpG.derecho_intel ERS_Ld5.ne30pg3_t232.B1850C_LTso.derecho_intel.allactive-decstart ERS_Ld5.ne30pg3_t232.BHISTC_LTso.derecho_intel.allactive-decstart ``` All tests passed and were bit-for-bit, except for this failure that also failed in the baseline: `FAIL NCK_Ly3.f09_g17_gris20.T1850Gg.derecho_gnu COMPARE_base_multiinst` (2) Full prealpha testing on derecho and izumi with comparison against cesm3_0_alpha08e Tests passed except for tests that also failed in the baselines or seemed to have passed due to tweaks made in the baseline (e.g., increasing wallclock time for `SUB_D_Ln9.ne3pg3_ne3pg3_mt232.FHIST.izumi_nag.cam-outfrq9s`). Baseline failures were all as expected (except that a few B compset tests failed baseline comparison due to a difference in which cpl.hx.ww3 files were present in the run vs. baseline, which presumably is due to something outside of this PR): ``` ERS_Ld5.ne30pg3_t232.B1850C_LTso.derecho_intel.allactive-decstart ERS_Ld5.ne30pg3_t232.BHISTC_LTso.derecho_intel.allactive-decstart ERS_Ly7.f09_g17_gris4.T1850Gg.derecho_intel MULTINOAIS_Ly2.f10_f10_ais8gris4_mg37.I1850Clm50SpRsGag.derecho_intel.cism-change_params SMS_Lm13.f10_f10_mg37.I1850Clm50SpG.derecho_intel SMS_D_Ly1.f09_g17_ais8.T1850Ga.derecho_gnu SMS_D_Ld5_P24x1.f10_f10_ais8gris4_mg37.I1850Clm50SpGag.izumi_nag.cism-test_coupling ``` Note that I showed that these differences go away when comparing against baselines generated with ESCOMP#633 , for similar or identical tests. So I think it's safe to conclude that this is bit-for-bit with ESCOMP#633 .
…comps Merging cime changes run through git-filter-repo to extract history of xcpl_comps_nuopc. The cime branch was created with: git filter-repo --path CIME/non_py/src/components/xcpl_comps_nuopc --path CIME/data/src/components/xcpl_comps_nuopc --path CIME/data/components/xcpl_comps_nuopc --path src/components/xcpl_comps_nuopc Note that this excludes the earlier history in src/components/xcpl_comps, because that history is messy and introduces a bunch of extra commits into history.
These don't appear to be used in a CESM build (they aren't present in any components other than these xcpl_comps)
This is copied exactly from CIME/build_scripts/buildlib.internal_components from ESMCI/cime@5444eae316
Add test components (X comps) ### Description of changes Add the test components that used to be in CIME. These can be used for testing CMEPS with artificial fields. In CESM, this is done with the `X` compset. These components belong more in CMEPS than in CIME. This brings in some, but not all history of these X components. I brought in as much history as I could fairly easily without going back to points where the history got very messy (with tons of merge commits that cluttered things up). Note that, for `X` compsets to continue to work, this should be coordinated with the CIME changes in ESMCI/cime#4946. But it's safe for the CMEPS changes to come in before the CIME changes. ### Specific notes Contributors other than yourself, if any: CMEPS Issues Fixed (include github issue #): Are changes expected to change answers? (specify if bfb, different at roundoff, more substantial) no - bfb Any User Interface Changes (namelist or namelist defaults changes)? ### Testing performed `SMS_Ld2_D_P8x1.f10_g37.X.green_gnu` in cesm3_0_alpha08f with cime and cmeps updated.
turn this debug alarm off by default
Collaborator
|
@mvertens did we check if the lnd2rof tracers still work with this? (Even though there is a known restart issue) |
Collaborator
Author
|
@mvdebolskiy - good point. I need to run the CTSM test suite to validate this - and it's been impossible to get any work done on betzy. We should not merge this PR until I am able to do more testing. |
Collaborator
|
@mvertens I can test this out, I need to update cdeps, ccs_config for CTSM-checkouts anyway. |
Collaborator
Author
|
@mvdebolskiy - that would be awesome. Thanks. I was not able to get anything done on either betzy or olivia today. Hopefully, this will work. Happy to help solve issues if it does not. |
mvdebolskiy
approved these changes
Apr 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of changes
This PR merges to the ESCOMP CMEPS1.1.41 as a next step in making merging ESCOMP and NorESM CMEPS development.
Note the majority of files for this PR come from the introduction of the new X compsets introduced in med_test_comps/ .
Specific notes
Contributors other than yourself, if any: None
CMEPS Issues Fixed: None
Are changes expected to change answers? Only for compsets with active CISM - all others are bfb
Any User Interface Changes? None
Testing performed
prealpha_noresm - compared to baselines created in testing NorESMhub/CDEPS#34 - all tests were bfb except those with active CISM