Python-ize the remaining bash run scripts, parallelize more tasks, fix some bugs

[mkavulich]

DESCRIPTION OF CHANGES:

While this touches a lot of files and changes a lot of lines, this is actually a fairly simple PR:

• The remaining bash/shell-based runscripts are converted to Python
• Parallelism functionality is added to new jobs.
• A few bugs are fixed

Bash --> Python conversion and parallelism

The remaining bash run scripts are now converted to Python. With this change, I have some demonstrations of the new functionality and cleanup that this allows:

1. The main reason this was done: now we can call multiple simultaneous instances of METplus from a single task using Python multiprocessing. This has been added for the GenEnsProd and EnsembleStat tasks, and should be added to more in the future. Several WE2E tests had parallelism added or increased. This resulted in significant speedups for many tests, especially MET_ensemble_verification_only_vx_time_lag that went from requiring more than 4 hours on Ursa to around 50 minutes, mostly due to speedup of the run_MET_GenEnsProd_vx_UPA and run_MET_GenEnsProd_vx_SFC tasks. Running all WE2E tests now takes just over an hour on Ursa, this could be sped up further by decreasing the --delay option and staging some obs data (Need to pre-stage some observations for WE2E tests to make them go quicker #30).
2. Most tasks now have their own dedicated config file section. This allows for easy specification of various job-specific variables in the config file, such as the number of parallel tasks for that job, its memory, and wall clock time, without requiring unwieldy long variable names to disambiguate. We can also organize the config sections into subsections: I plan to use this functionality heavily in the future, especially when tackling Incorporate the yaml files defining accumulation thresholds into the main config file #17
3. We can now use python native data types for things like lists rather than bash-like arrays that are hard to deal with. I made this conversion for ENS_TIME_LAG_HRS as an example, and it already removed the need for a python package (ast) and a few lines of logic.
4. Finally, a whole bunch of now-unused custom bash functions are removed.

Bugs fixed

• Put in checks to ensure that the number of tasks will always be an integer
• For parallel jobs, some METplus wrappers make use of a "staging directory"; we need to ensure that each parallel task has its own staging directory
• There was a discrepancy between the behavior of the legacy ush/set_vx_params.sh and the converted ush/set_vx_params.py that resulted in APCP and ASNOW verification tasks missing an h after the number of accumulated hours in both the job name and filenames for GridStat. This has been corrected.
• There was an occasional issue where setup_conda.sh would output a scary error message due to reading from a broken pipe (this would not cause any issues, but it could be concerning to the user). This is now fixed.
• GenEnsProd had a similar incorrect check for OBS_DIR to the one fixed for PCPCombine non-obs tasks (fixed in Addressing several issues: bug fixes and improvements #25). This is now fixed as well.

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • I have not removed any Bash functionality still in use through ush/launch_vx_wflow.sh and job scripts, so I believe there should be no loss of user-facing functionality.

TESTS CONDUCTED:

Ran all WE2E tests on Ursa. Used George's new regression testing functionality to compare the before-and-after performance and results. I confirmed all WE2E tests produced bit-for-bit identical results, with the only differences being in filenames due to the above-mentioned GridStat filename fix (I renamed all APCP and ASNOW GridStat output files manually to the old names in my test directory to ensure the regtest diff script would return "Success").

• derecho
• hera
• hercules
• orion
• ursa
• other (describe)

DEPENDENCIES:

• [develop]: Add regression testing #28 merged

DOCUMENTATION:

Documentation was not updated with this PR, but almost all user-facing functionality remains identical. Will need to do a final dedicated push to update documentation in a future PR (tracked in #4)

ISSUE:

• Partially resolves Need to reduce WE2E test runtimes #31
• Resolves Convert remaining run scripts from Bash to Python #8
• Resolves print_info_msg() bash function not working #15

CONTRIBUTORS:

A few commits in this PR were authored, partially or in full, by Claude CLI. These commits include the specific model used as a co-author. All of these changes were manually reviewed by me and checked for correctness.