ccpp-capgen v1: a unified successor to ccpp-prebuild and ccpp-capgen v0

[climbfuji]

Description

Note. ccpp-capgen v1 was previously called ccpp-capgen-ng. Some of the supporting documents attached below (until 2026-06-28) use this terminology.

ccpp-capgen v1 is a next-generation code generator for the CCPP framework. It is not a criticism of the existing tools — both ccpp-prebuild and ccpp-capgen v0 have served their respective communities well and embody years of careful design. This project attempts to synthesize the strengths of both: the performance of ccpp-prebuild and the metadata richness and feature set of ccpp-capgen v0, while also taking advantage of relaxed code-generator requirements that have emerged over time and lessons learned from operating both tools in development, research, and production. It also aims to resolve several outstanding issues that have been blocking progress toward a long-discussed unification of the two code generators.

Supporting documents — including a redesign analysis, the LLM prompt used to bootstrap development, developer and user documentation, and a migration guide — will be provided as part of the draft pull request.

The code was developed with the assistance of an LLM (Claude, by Anthropic) as a coding partner. Where possible, existing code was reused and repackaged.

Testing

Python docstring tests, unit tests

Significantly expanded test coverage with Python docstring tests and 1500-ish unit tests. Run in CI, all pass.

End-to-end tests

The following tests and features are currently supported by ccpp-capgen v1. They run successfullly in CI and on my development boxes.

• Former ccpp-capgen v0 tests:
  • advection
  • advection_auto_clone (support legacy constituents auto-cloning from metadata)
  • capgen_ng
  • ddthost
  • nested_suite
  • var_compat
• Former ccpp-prebuild tests:
  • chunked_data
  • blocked-data not supported in capgen or capgen-ng
  • opt_arg
• Newly added features/tests:
  • multi-instances
  • multi-instances + constituents
  • suite_allocate (exercises suite-owned variables with various types of runtime-known dimensions)
  • constituents_dim (suite- and host-owned variables with dimensions from constituents objects)

Host model integration

• CCPP-SCM all suites compile and all tests run, results are b4b identical for all except a few tests that are also problematic in develop
  • Transition CCPP framework from ccpp-prebuild to ccpp-capgen v1 ccpp-scm#695 and PRs listed in there
  • Replace ccpp-prebuild and ccpp-capgen v0 with ccpp-capgen v1 #762
  • Updates for ccpp-capgen v1 ccpp-physics#1221
  • Updates for ccpp-capgen v1 TEMPO#31
  • Metadata bug fixes ufs-community/c3#3
• CAM-SIMA Python unit tests pass, system tests pass with Intel and GNU
  • https://github.com/climbfuji/CAM-SIMA/tree/feature/capgen-ng
  • Feature/capgen ng climbfuji/CAM-SIMA#1
  • Feature/capgen ng climbfuji/atmospheric_physics#1
  • All regression tests (test/system) pass with both GNU and Intel, and with b4b-identical results except for two tests that also fail on development (SMS_Ln9.ne3pg3_ne3pg3_mg37.FKESSLER.derecho_intel.cam-outfrq_se_cslam_multitape, SMS_Ln9.ne3pg3_ne3pg3_mg37.FADIAB.derecho_gnu.cam-outfrq_se_cslam)
• UFS all regression tests pass, answers are bit-for-bit identical
  • Updates for ccpp-capgen v1 ufs-community/ufs-weather-model#3288
  • Updates for ccpp-capgen v1 NOAA-EMC/ufsatm#1125
  • Replace ccpp-prebuild and ccpp-capgen v0 with ccpp-capgen v1 #762
  • Updates for ccpp-capgen v1 ufs-community/ccpp-physics#378
  • Updates for ccpp-capgen v1 NOAA-GFDL/GFDL_atmos_cubed_sphere#419
  • Updates for ccpp-capgen v1 NOAA-EMC/CMEPS#173
• NEPTUNE complete
  • Pull requests merged in NRL Enterprise GitHub
  • All 300+ tests pass with the three mandatory compilers (Intel LLVM, GCC, LLVM native)
  • Regular, mid- and high-altitude physics covered (including proprietary NRL physics code)

Issues

A list of issues that will be addressed with this issue will be provided in the associated pull request.

Addressing this issue will also address/close the following issues if/when the participating host models switch to capgen-ng:

• Rethinking horizontal dimensions #760
• Weird Capgen fortran parsing error? #757
• Bug in capgen (test/capgen_test) ? surface_air_pressure outvar in ddt_suite #755
• Document --kind-type in ccpp_capgen.py CLI/help, and consider reversing the syntax for external kind modules #754
• Remove unused metadata attribute state_variable #753
• capgen: Add capability and test for passing mpi communicator as a flat field #751
• capgen: case-sensitivity when comparing metadata to Fortran #746
• Bug in scripts/file_utils.py / move_modified_files routine #744
• Unallocated fields passed into Group Caps #690
• Capgen performance when using many suites #688
• Add support for MPI_Comm (mpi08) object in capgen #686
• Transition UFS weather model to Capgen #682
• Transition CCPP-SCM to Capgen #680
• Transition NEPTUNE to capgen #679
• Enhanced/Expanded Testing  #595
• capgen's Fortran parser does not allow array bounds which are not Fortran identifiers #588
• Add "number of instances" dimension to module-level variables to support multi-instance usage #561
• capgen: check handling of type attribute for ddts #516
• Fortran/metadata comparison incorrectly reporting bug when comparing Fortran variable declarations and metadata for derived types #360
• CCPP Framework to provide API calls for suite inspection by host #380

To check/may not be implemented yet:

• capgen Fortran parser does not accept internal DDTs as part of a registered DDT #749
• capgen: Invalid character declaration, 'character(len=nf_file_length) ' #748
• capgen: dimension specifier foo(N) is not the same as foo(1:N) in Fortran code #747
• Errors when integer variable is output from one phase and used as an interstitial dimension in another #681

Working towards:

• Simplification of ccpp-framework code #737

Supporting Materials

See doc/ folder in the feature/capgen-v1 - documents evolved over time (commit history).

The documents were produced by Claude during development and most have them have not been formally reviewed. They are shared for transparency and to give context for the design decisions made. They should be treated as starting points rather than authoritative references.

An update of the CCPP technical documentation is made in NCAR/ccpp-doc#81.

[bbakernoaa]

@climbfuji I'm really interested to get this moving forward. How can I help move this forward?

[climbfuji]

@climbfuji I'm really interested to get this moving forward. How can I help move this forward?

@bbakernoaa Thank you very much for your offer to help, very much appreciated. There are many moving pieces. So far, this has been tested with CCPP-SCM and with NEPTUNE (with three compilers, Intel LLVM + GCC + LLVM native). The next tests need to cover UFS and CAM-SIMA. For CAM-SIMA, I'll work with the developers @peverwhee and @nusbaume. For UFS, your help would be appreciated, but we should coordinate with @dustinswales.

To prepare for UFS and SCM updates, we need to first fix the Metadata vs Fortran inconsistencies in the NCAR CCPP physics. I created pull requests for the UFS weather model, UFSATM, and CCPP physics ufs/dev for that (b4b identical w/ current baseline). It would be great if we could push these pull requests through quickly. When these changes come back to the authoritative NCAR ccpp-physics main branch, we can update the SCM and NEPTUNE branches. The changes will minimize the differences that we need to make when we transition to the new code generator. Maybe you can help with raising the priority for these PRs. I am also tagging @grantfirl and @dustinswales here.

[bbakernoaa]

@climbfuji I'm really interested to get this moving forward. How can I help move this forward?

@bbakernoaa Thank you very much for your offer to help, very much appreciated. There are many moving pieces. So far, this has been tested with CCPP-SCM and with NEPTUNE (with three compilers, Intel LLVM + GCC + LLVM native). The next tests need to cover UFS and CAM-SIMA. For CAM-SIMA, I'll work with the developers @peverwhee and @nusbaume. For UFS, your help would be appreciated, but we should coordinate with @dustinswales.

To prepare for UFS and SCM updates, we need to first fix the Metadata vs Fortran inconsistencies in the NCAR CCPP physics. I created pull requests for the UFS weather model, UFSATM, and CCPP physics ufs/dev for that (b4b identical w/ current baseline). It would be great if we could push these pull requests through quickly. When these changes come back to the authoritative NCAR ccpp-physics main branch, we can update the SCM and NEPTUNE branches. The changes will minimize the differences that we need to make when we transition to the new code generator. Maybe you can help with raising the priority for these PRs. I am also tagging @grantfirl and @dustinswales here.

@climbfuji Thanks. I would really like to get this moving forward as fast as possible. @dustinswales how can I help?

[climbfuji]

@climbfuji I'm really interested to get this moving forward. How can I help move this forward?

@bbakernoaa Thank you very much for your offer to help, very much appreciated. There are many moving pieces. So far, this has been tested with CCPP-SCM and with NEPTUNE (with three compilers, Intel LLVM + GCC + LLVM native). The next tests need to cover UFS and CAM-SIMA. For CAM-SIMA, I'll work with the developers @peverwhee and @nusbaume. For UFS, your help would be appreciated, but we should coordinate with @dustinswales.
To prepare for UFS and SCM updates, we need to first fix the Metadata vs Fortran inconsistencies in the NCAR CCPP physics. I created pull requests for the UFS weather model, UFSATM, and CCPP physics ufs/dev for that (b4b identical w/ current baseline). It would be great if we could push these pull requests through quickly. When these changes come back to the authoritative NCAR ccpp-physics main branch, we can update the SCM and NEPTUNE branches. The changes will minimize the differences that we need to make when we transition to the new code generator. Maybe you can help with raising the priority for these PRs. I am also tagging @grantfirl and @dustinswales here.

@climbfuji Thanks. I would really like to get this moving forward as fast as possible. @dustinswales how can I help?

Realized that Dustin is out of the office this week. I suggest we work with EPIC to prioritize the metadata-fortran bug fixes this week and early next week start working on dropping capgen-ng in the UFS.

@ligiabernardet FYI