[develop]: Add regression testing by georgemccabe · Pull Request #28 · dtcenter/dtc-vx-workflow

georgemccabe · 2026-06-30T20:26:03Z

DESCRIPTION OF CHANGES:

Note: This PR contains changes from PR #25 to address some issues running the tests. Until that PR is merged, the differences in this PR only can be viewed from this link:
feature/fix_some_issues...feature/regression_tests

Adds files in tests/regression directory
- run_regression.py - run all end-to-end tests for a given branch/commit, pull request, or the "baseline" (commit/branch that is considered to be the truth data to compare new test results)
- run_diff.py - Compare output generated by run_regression.py to other output to see if there are any differences using the METplus diff util
- README.md - instructions and examples to run regression test scripts
- regression_baseline.py - stores branch (develop) and commit that is considered the baseline to compare new results. If a PR will changes these results and is expected, the value in this file should be updated to the latest commit of the develop branch after the PR has been merged.
- regression_common.py - variables/functions common to multiple scripts, e.g. the default location of the regression test output and the name of this repo
- environment.yml - conda environment file to create vx_diff environment used to run the run_diff.py script
Updated setup_conda.sh script to take an argument vx_diff which creates/activates the diff environment instead of the vx_workflow environment

An example of the log file created by the diff script can be found on ursa:
/scratch3/BMC/dtc/George.Mccabe/dtc-vx-workflow_testing/diff_WE2E_20260630200405.txt

Type of change

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

TESTS CONDUCTED:

DEPENDENCIES:

DOCUMENTATION:

ISSUE:

CHECKLIST

My code follows the style guidelines in the Contributor's Guide
I have performed a self-review of my own code using the Code Reviewer's Guide
I have commented my code, particularly in hard-to-understand areas
My changes need updates to the documentation. I have made corresponding changes to the documentation
My changes do not require updates to the documentation (explain).
My changes generate no new warnings
New and existing tests pass with my changes
Any dependent changes have been merged and published

LABELS (optional):

A Code Manager needs to add the following labels to this PR:

CONTRIBUTORS (optional):

…ent branch and latest commit

…is currently written to be run from that directory

…anch directory before running tests

…und by end-to-end test runs

…ed up test runs. Clean up logic to report if a test dir is not found in either baseline or new output

- Replace call to load_modules_wflow.sh with call to setup_conda.sh - Since we no longer have module loads, ROCOTO_PATH is provided as a variable in the machine file for each platform

…king yet

… does this automatically

Read ROCOTO_PATH from the machine config file and prefix all calls to rocotorun and rocotostat with it, consistent with launch_vx_wflow.sh. Machines without ROCOTO_PATH (jet, noaacloud, linux, macos) fall back to bare binary names. rocoto_path is stored in the WE2E monitor YAML and propagated through update_expt_status/compare_rocotostat; added to all non-task entry skip lists to avoid being treated as a task. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

The statuses loop in update_expt_status had 8-space indentation while the other two skip lists had 12-space indentation, so the replace_all edit missed it. This caused a TypeError when iterating over the expt dict since rocoto_path (a string) was not excluded from the task loop. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Resolve SCRIPT_DIR from ${BASH_SOURCE[0]} at the top and anchor all relative file/directory references (conda_loc, conda/, environment.yml) to it. Also replaces readlink -f with cd && pwd for macOS compatibility without requiring GNU coreutils. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…p_conda.sh script

…ut files that were skipped or had no differences

…ng the rocoto exes

…write permissions set so multiple users can run tests in that area

…ent <native> tag for rocoto when submitting to the HPSS queue (just like how get_obs tasks already do)

…_tests

mkavulich

Looks good, I have just a few comments/suggestions.

mkavulich · 2026-07-13T17:54:41Z

+
+The default behavior is to run the diff utility on all files in the dated subdirectories
+under the output directory, because these are assumed to contain the actual MET output.
+The `--diff_all_files` argument can be added to run the diff utility on all files in the output directory.


I'm a bit confused about the behavior of this flag. It seems like the SKIP_KEYWORDS files are still skipped even when this flag is enabled...is that the intended behavior? If so, I think this flag needs a better name, since it implies that all files will be diffed when active.

It would also be nice if there was a flag to allow you to run a diff where it ignores missing test directories...for example, if you only run a subset of all tests, but still want to check them for success against the baseline.

Regarding the --diff_all_files argument, I agree that it is a little misleading. The default behavior is to run the diff on each YYYYMMDDHH directory under each workflow directory, and the --diff_all_files argument runs the diff on each workflow directory, which will include some of the other files, like input files that are downloaded.

You are correct that the SKIP_KEYWORDS are still used if the flag is set or not. Excluding this list of skip keywords would result in a lot of misleading diffs being flagged in both configurations.

Any suggestions for improving this? Maybe the --diff_all_files argument is not really needed if we are only concerned with comparing the MET output? I added this option with the intention of being able to see if there are differences in the input files that could change at the source unexpectedly, which would explain the differences in the output. Maybe this is not a common occurrence and unnecessary to easily check?

Regarding an option to run a subset of tests:

This is a good idea. I could add an optional --tests argument for the run_regression.py script to specify a subset of tests that could be passed to the run_we2e_tests.py script to tell it to run that subset. Currently it just calls it with --tests all.

A corresponding argument could be added to the run_diff.py script to determine which sub-directories (tests) to run the diff on. This would require pulling out the logic that parses the --tests argument in run_we2e_tests.py so it could be used by the run_diff.py script in the same way. Thoughts?

With regards to the --diff_all_files option, I think renaming it to --diff_obs or --diff-input or something similar would suffice.

I think utilizing "tests" logic from run_we2e_tests.py sounds great.

@mkavulich, I renamed the --diff_all_files argument to --diff_inputs to run_diff.py and added the --tests argument to run_regression.py to run a subset of tests by passing the value of the argument directly to run_we2e_tests.py.

I am working on updating the run_diff.py to take a --tests argument to run the diff logic on a subset of tests in another branch. These changes are a little more involved to share the functions to gather/check the e2e test names and isolate the vx_workflow and vx_diff dependencies, so I suggest that those changes be added to a separate PR. Does that sound good to you?

Co-authored-by: Michael Kavulich <kavulich@ucar.edu>

… script for clarity

… subset of tests (defaults to all tests)

mkavulich

Thanks for all your work on this

georgemccabe and others added 30 commits May 28, 2026 10:29

add staged data directory path for ursa

6588c25

initial version of regression test script -- just runs tests for curr…

0838daf

…ent branch and latest commit

fix failures due to incorrect type for maxtries values

df0569b

fix to allow we2e tests to run

d77a41c

create python script to handle running regression test for branch or PR

3ab80ad

handle Path vs string in git commands

91fc2de

fix repo location

e1f13cf

run script from tests/WE2E dir from branch because run_we2e_tests.py …

600a67d

…is currently written to be run from that directory

update version of uwtools

cbf9a14

updates to fix running tests via subprocess to set up conda env in br…

582894f

…anch directory before running tests

rearrange functions in order of use and remove whitespace

4f4bbee

force install conda env into local conda directory to ensure it is fo…

3c4980d

…und by end-to-end test runs

support creating vx_diff environment if vx_diff argument is provided

daa8991

added environment file for conda env to run METplus diff_util

b6d1c3c

fix setting of env yaml file after dependent env var is set

1cc4300

start of script to run METplus diff util on each experiment's output

9197949

Add option to diff all files or only diff dated subdirectories to spe…

1ace70a

…ed up test runs. Clean up logic to report if a test dir is not found in either baseline or new output

fix handling of success status

3730b4c

print details of test results

7616bec

clean up import

8eb20ea

Add staged location data for Ursa

dfc5cf7

Fix issue with launching jobs from cron

f079f0f

Fix setup_conda for environments where LD_LIBRARY_PATH is unset

894de82

Fix issues with crontab submission:

4514341

- Replace call to load_modules_wflow.sh with call to setup_conda.sh - Since we no longer have module loads, ROCOTO_PATH is provided as a variable in the machine file for each platform

Attempting to get automated pulling of best track data; not quite wor…

8f6b232

…king yet

borrow latest conda setup logic from mpas_plot repository

76567bc

Wording fixes, suppress "conda activate" message since setup_conda.sh…

abae915

… does this automatically

mkavulich and others added 16 commits June 23, 2026 17:41

Completely disable retrieve_data.py test

0666859

Merge branch 'develop' into feature/fix_some_issues

09e095b

Merge branch 'develop' into feature/regression_tests

6cede53

merged feature/fix_some_issues branch and resolve conflicts with setu…

abd32fc

…p_conda.sh script

add logging options

6be4b6b

fix list of all tests (union not intersection)

adef994

print path to log file to terminal

9583f1e

add --debug command line argument to produce more log information abo…

1f85c91

…ut files that were skipped or had no differences

log paths that are compared by diff script

5f85da0

update the info about how to update the baseline used for the diff

f96adec

remove module load of rocoto since the changes in PR #25 handle findi…

7c32a83

…ng the rocoto exes

add instruction to ensure shared regression test directory has group …

68ca84e

…write permissions set so multiple users can run tests in that area

Only check for OBS_DIR if obs are needed

612f3ac

Better check: dont check for OBS_DIR for GenEnsProd task

87db0b4

Fix error submitting TCPAIRS job on Hera: we need to specify a differ…

827a3e8

…ent <native> tag for rocoto when submitting to the HPSS queue (just like how get_obs tasks already do)

Merge remote-tracking branch 'origin/develop' into feature/regression…

5f021c6

…_tests

mkavulich marked this pull request as ready for review July 9, 2026 21:19

mkavulich requested review from EdwardSnyder-NOAA, JeffBeck-NOAA and gsketefian as code owners July 9, 2026 21:19

mkavulich mentioned this pull request Jul 10, 2026

Python-ize the remaining bash run scripts, parallelize more tasks, fix some bugs #33

Open

9 tasks

mkavulich reviewed Jul 13, 2026

View reviewed changes

georgemccabe and others added 3 commits July 15, 2026 11:35

Add stage directory to diff ignore list

23c38f0

Co-authored-by: Michael Kavulich <kavulich@ucar.edu>

per PR feedback from @mkavulich, rename command line argument to diff…

1917126

… script for clarity

per PR feedback from @mkavulich, add argument to test script to run a…

606a882

… subset of tests (defaults to all tests)

georgemccabe marked this pull request as draft July 16, 2026 21:50

georgemccabe marked this pull request as ready for review July 17, 2026 15:49

mkavulich approved these changes Jul 17, 2026

View reviewed changes

mkavulich merged commit da686ac into develop Jul 17, 2026
3 checks passed

georgemccabe deleted the feature/regression_tests branch July 17, 2026 20:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[develop]: Add regression testing - #28

[develop]: Add regression testing#28
mkavulich merged 74 commits into
developfrom
feature/regression_tests

georgemccabe commented Jun 30, 2026 •

edited

Loading

Uh oh!

mkavulich left a comment

Uh oh!

Uh oh!

mkavulich Jul 13, 2026

Uh oh!

mkavulich Jul 13, 2026 •

edited

Loading

Uh oh!

georgemccabe Jul 15, 2026

Uh oh!

georgemccabe Jul 15, 2026

Uh oh!

mkavulich Jul 16, 2026

Uh oh!

georgemccabe Jul 17, 2026

Uh oh!

mkavulich left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

georgemccabe commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

DESCRIPTION OF CHANGES:

Type of change

TESTS CONDUCTED:

DEPENDENCIES:

DOCUMENTATION:

ISSUE:

CHECKLIST

LABELS (optional):

CONTRIBUTORS (optional):

Uh oh!

mkavulich left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mkavulich Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

mkavulich Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

georgemccabe Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

georgemccabe Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

mkavulich Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

georgemccabe Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

mkavulich left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

georgemccabe commented Jun 30, 2026 •

edited

Loading

mkavulich Jul 13, 2026 •

edited

Loading