[develop]: Regression test improvements#39
Open
georgemccabe wants to merge 90 commits into
Open
Conversation
…ent branch and latest commit
…is currently written to be run from that directory
…anch directory before running tests
…und by end-to-end test runs
…ed up test runs. Clean up logic to report if a test dir is not found in either baseline or new output
- Replace call to load_modules_wflow.sh with call to setup_conda.sh - Since we no longer have module loads, ROCOTO_PATH is provided as a variable in the machine file for each platform
… does this automatically
Read ROCOTO_PATH from the machine config file and prefix all calls to rocotorun and rocotostat with it, consistent with launch_vx_wflow.sh. Machines without ROCOTO_PATH (jet, noaacloud, linux, macos) fall back to bare binary names. rocoto_path is stored in the WE2E monitor YAML and propagated through update_expt_status/compare_rocotostat; added to all non-task entry skip lists to avoid being treated as a task. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The statuses loop in update_expt_status had 8-space indentation while the other two skip lists had 12-space indentation, so the replace_all edit missed it. This caused a TypeError when iterating over the expt dict since rocoto_path (a string) was not excluded from the task loop. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Resolve SCRIPT_DIR from ${BASH_SOURCE[0]} at the top and anchor all
relative file/directory references (conda_loc, conda/, environment.yml)
to it. Also replaces readlink -f with cd && pwd for macOS compatibility
without requiring GNU coreutils.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ng the rocoto exes
…write permissions set so multiple users can run tests in that area
…efactor to reduce duplication and cognitive complexity
…ent <native> tag for rocoto when submitting to the HPSS queue (just like how get_obs tasks already do)
Co-authored-by: Michael Kavulich <kavulich@ucar.edu>
… script for clarity
… subset of tests (defaults to all tests)
…n be imported and functions that don't require the extra dependencies can be used
…s so it can be used by the run_diff regression test script
… to match run_we2e_tests.py functionality
…ss. List of tests to process is used to build list of diff tests to perform so it will report if any of those tests are missing from either output directory. Rename --baseline argument to --baseline_dir to distinguish it from --baseline argument to run_regression.py script which does not do the same thing
…written there instead of tests/WE2E directory
…o check for the tests/WE2E full path
…resolve '..' characters in paths
…esn't need to be set when calling pylint
georgemccabe
marked this pull request as ready for review
July 21, 2026 20:20
georgemccabe
requested review from
EdwardSnyder-NOAA,
JeffBeck-NOAA and
gsketefian
as code owners
July 21, 2026 20:20
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DESCRIPTION OF CHANGES:
--testsargument ofrun_we2e_tests.pyinto python_utils so that it can be used by therun_diff.pyregression test script to determine which subdirectories to diff--testsargument parsing (identified unsupported inputs that could be added later)--testsargument to run a subset of tests and diff a subset of tests (reports failure if requested test was not generated by either run)run_we2e_tests.pyscript so that it can be run from any directory instead of just the tests/WE2E directory (allows log outputs to be written to more appropriate location when running via regression test scriptsrun_diff.pyargument--baselineto--baseline_dirto differentiate from the `run_regression.py argument (one says to run tests on the baseline commit while the other overrides the location of the baseline data)run_regression.pyscript where--testsargument cannot take more than 1 input value (list of tests)Type of change
TESTS CONDUCTED:
On Ursa
Ran regression test on this PR (39)
python3 ./tests/regression/run_regression.py --pr 39 --account dtc --machine ursaLog files:
Ran diff logic comparing PR 39 to baseline (feature/regression_tests/output.827a3e8)
No difference were found
Ran diff logic comparing same test output but only running the diff on a subset of tests (ensemble)
python3 ./tests/regression/run_diff.py ${test_dir} --tests ensemble/scratch3/BMC/dtc/dtc-vx-workflow_testing/diff_WE2E_20260721201839.txt
DEPENDENCIES:
None
DOCUMENTATION:
ISSUE:
LABELS (optional):
A Code Manager needs to add the following labels to this PR:
CONTRIBUTORS (optional):