Increase flexibility for multi-model ensembles, add REFS config file#38
Draft
mkavulich wants to merge 43 commits into
Draft
Increase flexibility for multi-model ensembles, add REFS config file#38mkavulich wants to merge 43 commits into
mkavulich wants to merge 43 commits into
Conversation
Converts all four remaining shell scripts in scripts/ to Python, completing the bash-to-Python migration: - check_post_output.py: calls set_leadhrs() to verify post-processed forecast output files exist within the allowed missing-file threshold - gridstat_or_pointstat_ensmean.py: GridStat/PointStat on ensemble mean output from GenEnsProd - gridstat_or_pointstat_ensprob.py: GridStat/PointStat on ensemble probability output from GenEnsProd - genensprod_or_ensemblestat.py: GenEnsProd or EnsembleStat with per-member forecast template list; GenEnsProd skips obs file check - pcpcombine.py: PcpCombine for FCST or OBS with double set_leadhrs call pattern; includes PM2.5/PM10 USER_DEFINED combine logic for AIRNOW Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
grep -q exits immediately on first match, closing the pipe before conda env list finishes writing. Conda's Python runtime catches the resulting SIGPIPE and prints a noisy error report to stderr. Adding 2>/dev/null suppresses it without affecting the grep result or exit code. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…s requires a string-subst dictionary in check_post_output.py; should be able to get rid of this logic in all scripts by the end. - Convert ENS_TIME_LAG_HRS to a true Python list. This removes need for strange "literal_eval" logic in gridstat_or_pointstat.py
…t; almost working
…h for ACCUM_HH definitions
…e an ensemble member index and some don't, so we need to add this as another optional variable to set_job_env.sh
…tion: section with tasks_per_node, walltime, and memory, as well as a TASKS: variable. Update some WE2E tests to use more parallelism
…execution: section with tasks_per_node, walltime, and memory, as well as a TASKS: variable, and apply memory and task settings to workflow definition file'
…place with printf
…_dir_file(), check_var_valid_value(), get_elem_inds(), is_element_of(), print_input_args(), and process_args()
…meter in single location.
…ENSMEM_NAMES which it turns out is deprecated!
…l simplify a lot of logic with the use of a list of filenames This commit also prepares to deprecate FCST_SUBDIR_TEMPLATE; this will also simplify logic
…s to add lines directly to that task's conf file. This is only for genensprod for now, but should be expanded to all tasks once every task has its own config section Additionally, more QOL improvements for REFS experiment: - Make OMP_NUM_THREADS configurable for MODE, GenEnsProd, and EnsembleStat - New task-specific section for pb2nc, make tasks and memory configurable; PB2NC now parallelized - Add VX_TASKS parallelism for gridstat_or_pointstat_ensmean.py and gridstat_or_pointstat_ensprob.py; need to add task-specific sections for these scripts in the future!
- Split out all "get_obs" tasks into their own taskgroup file. verify_pre.yaml is now split into get_obs.yaml and preprocessing.yaml - Remove taskgroups: definition from WE2E tests (except for MET_verification_only_vx, just to catch any future changes) - Create new sections for ensemble gridstat and pointstat tasks - Remove premature KeyError from config checks
…he "NUM_MISSING_OBS_FILES_MAX" functionality
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DESCRIPTION OF CHANGES:
NOTE: This PR is in draft mode because it contains the changes from #33, and should not be merged until after that one is merged.
You can view the changes for only this PR here: feature/pythonize_remaining_bash...feature/increase_flexibility
This PR represents the first round of changes needed for REFS ensemble experiments. The major changes are:
FCST_FN_TEMPLATEis now a list of files rather than a string. A string is still accepted in the config file for simplicity and back-compatibility.FCST_SUBDIR_TEMPLATEvariable is deprecated (any subdirectories needed here can be specified as part ofFCST_FN_TEMPLATE).verify_pre.yamltaskgroup is split into twoget_obs.yaml, which contains all of the obs-pulling taskspreprocessing.yaml, which contains the remainder of the preprocessing tasksThis allows users to easily just pull the observations they need without running any more tasks
pb2nc, as well as "Ensemble" GridStat and PointStat tasks now have their own dedicated config section, and can be parallelizedconf:is added to the GenEnsProd task; this allows the user to specify exact lines to add to a METplus conf fileush/configssubdirectory is created, which should be used to stage config files used for DTC experiments. I included a "first attempt" REFS config file asush/configs/SFE_2026_ensembles/config.firsttest.yaml, that is not perfect but serves as a proof of conceptSome other minor improvements are also included:
maxtriesvariable is now configurable for tasks that have their own config section. The default is changed to 1 for all of these tasks as wellselect_validtime_obs.pyno longer raises an exception when a missing file is encountered; this should be handed inget_obs.pyaccording to theNUM_MISSING_OBS_FILES_MAXsetting.rocotorunis reduced from 5 seconds to 1 second (except for Derecho which still requires an extended delay due to cacheing of job statuses)RUN_CMD_FCST,RUN_CMD_POST,RUN_CMD_PRDGEN,RUN_CMD_SERIAL,RUN_CMD_UTILS, andPRE_TASK_CMDS) are removed from config_defaults.yaml and the machine files, and one missing one from the machine files (SCHED_NATIVE_CMD_HPSS) is added to config_defaults.yamlVERBOSEis now false by defaultNUM_ENS_MEMBERSmust now always be a positive integer (1 for deterministic runs, default)uppercase()andlowercase()are removedType of change
verify_pre.yamltoget_obs.yamlandpreprocessing.yamlFCST_SUBDIR_TEMPLATEmust instead prepend that value toFCST_FN_TEMPLATE.TESTS CONDUCTED:
Regression testing using #33 as a baseline showed no changes in WE2E tests.
DEPENDENCIES:
DOCUMENTATION:
Will be handled later, tracked in #4
ISSUE: