Replace ccpp-prebuild and ccpp-capgen v0 with ccpp-capgen v1 - #762
Replace ccpp-prebuild and ccpp-capgen v0 with ccpp-capgen v1#762climbfuji wants to merge 81 commits into
Conversation
… feature/capgen-ng
…gen-ng: entire constituent state now per instance
…ts (#764) This PR targets the feature/capgen-v1 branch with follow-up and cleanup changes from manual code reviews (capgen/metadata/parse-tools/*) plus a few selected, minor improvements. This branch is being used by the CAM-SIMA feature branches that replace the original capgen v0 with the new capgen v1. - Temporary: run CI tests also for pull requests against branch feature/capgen-v1. - Simplify capgen/metadata/parse_tools/parse_source.py. Some of the removed features were (ab)used by CAM-SIMA and are now in the updated capgen compatibility layer in the CAM-SIMA feature branch. - Simplify capgen/metadata/parse_tools/parse_log.py. Requires a tiny change in the CAM-SIMA compatibility layer (now in the (CAM-SIMA feature branch). - Add documentation, simplify, and add a comment about a latent bug that doesn't hurt is today w.r.t. escaping periods in decimals in capgen/metadata/parse_tools/fortran_conditional.py. - capgen/metadata/parse_tools/io_helpers.py: better documentation, preserve default umask. - Simplify capgen/metadata/parse_tools/xml_tools.py: remove unneeded arguments, trim down docstring tests (covered in unit tests and end-to-end tests). Requires small changes in the CAM-SIMA compatibility layer (now in the (CAM-SIMA feature branch). - Strip down capgen/metadata/parse_tools/parse_checkers.py, update capgen/metadata/metadata_table.py accordingly. - Not cleanup: Fix the trailing comments after the variable conversions in the Group cap ([feature/capgen-v1-cleanup-01] Add correct comment to supported variable conversions (#767) + bug fixes + unit test updates #768) by @dustinswales. - Not cleanup: Add additional report option --suite-variables to retrieve suite-owned variables from datatable. - Not cleanup: Log variable transformations during capgen call at default logging level (0d48ab0). --------- Co-authored-by: Courtney Peverley <courtneyp@ucar.edu> Co-authored-by: Dustin Swales <dustin.swales@noaa.gov>
Format update of `feature/capgen-v1` from `develop` to resolve conflicts after merging #765 into develop / #767 into `feature/capgen-v1`. There's a small leftover change in `end-to-end-tests/advection_auto_clone/cld_liq.F90` that was made in #765 but not in #767. The other test data change in `test/advection_test/test_host_data.F90` (see https://github.com/NCAR/ccpp-framework/pull/765/changes#diff-04bf1167afd493d2ea37bd03d093e200ba8b563383bfd1cca3fc9126aca9c4a9) is already in `feature/capgen-v1`. Co-authored-by: Courtney Peverley <courtneyp@ucar.edu>
… feature/capgen-v1
gold2718
left a comment
gold2718
left a comment
There was a problem hiding this comment.
I am having trouble giving this (moving target) a thorough review but have a question and a couple of concerns. The question is about the protected attribute (see below).
Concern 1: By removing and changing the old unit tests, it is harder to verify that the parser / generator works correctly. How do you verify test coverage relative to the earlier (admittedly not great) version? Are all the tests functionally replicated?
Concern 2: Validation of metadata against the Fortran source has been relegated to capgen/ccpp_validator.py. While running this tool can be built into the build process, the old version ran on the already parsed metadata. At best, this will slow down the validation process as the metadata has to be re-parsed. However, is the metadata in fact parsed the same? Is there a reason this could not be an optional part of capgen that works on the parsed metadata?
| source = 'suite' | ||
| elif host_entry is None and suite_var is None: | ||
| # Case 2 or 3. | ||
| if intent == 'out': |
There was a problem hiding this comment.
I may have missed it, but is the protected attribute checked anywhere? If not, can you introduce a check for host_entry.protected and intent in ('out', 'inout')?
There was a problem hiding this comment.
I'll get back to you on that one.
I hope that no more updates are needed for now. The most recent update to CAM-SIMA necessitated a number of changes and my limited bandwidth (technically on PTO) didn't allow me to create yet another set of branches and PRs. Future changes required due to CAM-SIMA updates will need to go in yet another staging branch to avoid updating the
This is indeed the case. The existing tests from capgen v0 and prebuild were copied over 1:1, then the validation trimmed down to just the Fortran part where applicable. The blocked-data test for prebuild was removed, because we never intended to support blocked data structures. New tests were added for new functionality (e.g. multi-instance). An additional advection test with the now-deprecated "auto-clone constituents" (= defined in metadata only) enabled via a legacy shim. Several passes were done with the LLM to ensure that the current tests cover everything the old tests did, plus more. But also to remove duplicates between unit tests and docstring tests. [Edit] In addition to an already better test coverage, we benefit greatly from the integration with the four host models. These exercise many different aspects of capgen, and the fact that the regression tests all pass and are bit-for-bit identical for CAM-SIMA, CCPP-SCM, NEPTUNE, and UFS, provides us with real evidence that the features that matter (i.e. that are actually being used) are correct.
This was a deliberate decision. A separation of concerns means that ccpp_capgen doesn't (need to) know anything about parsing Fortran code and can rely on metadata only. Separating these two steps also enables offline validation of metadata vs fortran without the need for a host model, and it's easier to see immediately if the validation step fails as part of the cmake build integration, or if it is the actual cap generation. Yes, the metadata is read twice in this approach, but the measured runtimes speak for themselves. For all 58 suites of CCPP-SCM, these are the runtimes:
It's cleaner, easier to understand, faster, and more aligned with modern software development practices (separation of concerns). |
2 participants
Description
This PR replaces the two code generators
ccpp-prebuildandccpp-capgen(version 0) with the new, unifiedccpp-capgen(version 1). It supports all host models (CAM-SIMA, CCPP-SCM, NEPTUNE, UFS) and combines the features fromccpp-capgen v0with the performance and operational robustness ofccpp-prebuild.This pull request will be updated continuously with links to updates of the CCPP technical documentation and additional information. See also #759 for background information and further materials.
User interface changes?: Yes (see documentation)
This feature branch
feature/capgen-v1will support the threeccpp-prebuild-based models CCPP-SCM, NEPTUNE, and UFS, which will transition toccpp-capgen v1before CAM-SIMA. Once CAM-SIMA transitions, this PR will be merged andfeature/capgen-v1will be locked.Todo
doc/need to be reviewed and used to update the existing CCPP technical documentation (https://github.com/ncar/ccpp-doc). Once that process is complete, we can remove or trim down the additional documentation indoc/.Issues
scripts/file_utils.py/move_modified_filesroutine #744To check/may not be implemented yet:
Testing
See #759