Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
b4b2764
Convert remaining bash ex-scripts to Python
mkavulich Jun 25, 2026
9e84507
Suppress broken pipe error from conda env list in setup_conda.sh
mkavulich Jul 2, 2026
b9a1112
- Update CHECK_POST_OUTPUT.sh to call new python script. For now thi…
mkavulich Jul 6, 2026
110875c
Converting jobs/GENENSPROD_OR_ENSEMBLESTAT.sh to use new python scrip…
mkavulich Jul 6, 2026
0d1082d
Found error: set_vx_params.py was not consistent with set_vx_params.s…
mkavulich Jul 6, 2026
363e15f
Update PCPCOMBINE.sh to use pcpcombine.py. Some PCP combine tasks hav…
mkavulich Jul 6, 2026
75d66cc
All run scripts pythonized!
mkavulich Jul 7, 2026
1067176
Update new genensprod_or_ensemblestat.py for parallel task submission
mkavulich Jul 7, 2026
57283a6
Add gridstat: and pointstat: config sections, give them each an execu…
mkavulich Jul 7, 2026
18aa8d9
For tools that use a staging directory, need a different one for each…
mkavulich Jul 8, 2026
b7477d0
Add genensprod: and ensemblestat: config sections, give them each an …
mkavulich Jul 8, 2026
2d74ee9
Only check for obs_dir for ensemblestat
mkavulich Jul 8, 2026
ddf8f8b
Ensure that numprocs is always an integer
mkavulich Jul 8, 2026
6314e52
Remove unused/deprecated shell scripts
mkavulich Jul 10, 2026
7746d97
Deprecate and remove "echo_lowercase" and "echo_uppercase" bash funct…
mkavulich Jul 10, 2026
562d342
Deprecate and remove print_info_msg() bash function per Issue #15, re…
mkavulich Jul 10, 2026
cf20e25
Deprecate and remove more unneeded bash functions: check_for_preexist…
mkavulich Jul 10, 2026
0654972
Remove the remaining unused bash functions: is_array() and create_sym…
mkavulich Jul 10, 2026
84e9e20
Add explicit !int tags for all TASKS: entry in config_defaults.yaml
mkavulich Jul 10, 2026
8be5b48
Make sure one more TASKS entry is an int
mkavulich Jul 13, 2026
99175bc
More processors!
mkavulich Jul 13, 2026
87613b5
Add checks for renamed/moved config sections in setup.py
mkavulich Jul 17, 2026
f5cffd2
Missed moving gridstat/pointstat settings to new section for MODE_AOD…
mkavulich Jul 17, 2026
6ef4227
Fix linting issues with ush/setup.py
mkavulich Jul 17, 2026
2b426c9
Fix rest of the linting
mkavulich Jul 17, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions jobs/ASCII2NC_OBS.sh
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ scrfunc_fp=$( $READLINK -f "${BASH_SOURCE[0]}" )
scrfunc_fn=$( basename "${scrfunc_fp}" )
scrfunc_dir=$( dirname "${scrfunc_fp}" )

print_info_msg "

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Per #15, this function has been deprecated. printf gives more analogous functionality to the old function (which called printf internally), particularly with regards to newlines, so using that instead of echo

printf "
========================================================================
Entering script: \"${scrfunc_fn}\"
In directory: \"${scrfunc_dir}\"
Expand All @@ -64,5 +64,5 @@ python $SCRIPTSdir/ascii2nc_obs.py ${VERBOSE_FLAG} \
--cycle_date="${YYMMDD}${HH}" \
--obtype="${OBTYPE}" || \
print_err_msg_exit "\
Call to \"ascii2nc_obs.sh\" from \"${scrfunc_fn}\" failed."
Call to \"ascii2nc_obs.py\" from \"${scrfunc_fn}\" failed."

12 changes: 9 additions & 3 deletions jobs/CHECK_POST_OUTPUT.sh
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,8 @@ sections=(
for sect in ${sections[*]} ; do
source_yaml ${GLOBAL_VAR_DEFNS_FP} ${sect}
done
# Sets up PYTHONPATH and VERBOSE environment variables
. $USHdir/set_job_env.sh
#
#-----------------------------------------------------------------------
#
Expand All @@ -52,17 +54,21 @@ scrfunc_fp=$( $READLINK -f "${BASH_SOURCE[0]}" )
scrfunc_fn=$( basename "${scrfunc_fp}" )
scrfunc_dir=$( dirname "${scrfunc_fp}" )

print_info_msg "
printf "
========================================================================
Entering script: \"${scrfunc_fn}\"
In directory: \"${scrfunc_dir}\"
========================================================================"
#
# Call the run script
#
$SCRIPTSdir/check_post_output.sh || \
python $SCRIPTSdir/check_post_output.py ${VERBOSE_FLAG} \

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Values that were previously read in from environment variables in the bash scripts are now explicitly passed as command-line arguments

--config="${GLOBAL_VAR_DEFNS_FP}" \
--cycle_date="${YYMMDD}${HH}" \
${ENSMEM_ARG} || \
print_err_msg_exit "\
Call to script \"check_post_output.sh\" from \"${scrfunc_fn}\" failed."
Call to \"check_post_output.py\" from \"${scrfunc_fn}\" failed."

#
#-----------------------------------------------------------------------
#
Expand Down
17 changes: 14 additions & 3 deletions jobs/GENENSPROD_OR_ENSEMBLESTAT.sh
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,8 @@ sections=(
for sect in ${sections[*]} ; do
source_yaml ${GLOBAL_VAR_DEFNS_FP} ${sect}
done
# Sets up PYTHONPATH, ACCUM_ARG, and VERBOSE environment variables
. $USHdir/set_job_env.sh
#
#-----------------------------------------------------------------------
#
Expand All @@ -48,15 +50,24 @@ scrfunc_fp=$( $READLINK -f "${BASH_SOURCE[0]}" )
scrfunc_fn=$( basename "${scrfunc_fp}" )
scrfunc_dir=$( dirname "${scrfunc_fp}" )

print_info_msg "
printf "
========================================================================
Entering script: \"${scrfunc_fn}\"
In directory: \"${scrfunc_dir}\"
========================================================================"
#
# Call the run script
#
$SCRIPTSdir/genensprod_or_ensemblestat.sh || \
python $SCRIPTSdir/genensprod_or_ensemblestat.py ${VERBOSE_FLAG} \
--config="${GLOBAL_VAR_DEFNS_FP}" \
--cycle_date="${YYMMDD}${HH}" \
--field_group="${FIELD_GROUP}" \
--obs_dir="${OBS_DIR}" \
--obtype="${OBTYPE}" \
--fcst_level="${FCST_LEVEL}" \
--fcst_thresh="${FCST_THRESH}" \
--metplus_tool="${METPLUSTOOLNAME}" \
${ACCUM_ARG} || \
print_err_msg_exit "\
Call to \"genensprod_or_ensemblestat.sh\" from \"${scrfunc_fn}\" failed."
Call to \"genensprod_or_ensemblestat.py\" from \"${scrfunc_fn}\" failed."

7 changes: 4 additions & 3 deletions jobs/GET_VERIF_OBS.sh
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ scrfunc_fp=$( $READLINK -f "${BASH_SOURCE[0]}" )
scrfunc_fn=$( basename "${scrfunc_fp}" )
scrfunc_dir=$( dirname "${scrfunc_fp}" )

print_info_msg "
printf "
========================================================================
Entering script: \"${scrfunc_fn}\"
In directory: \"${scrfunc_dir}\"
Expand All @@ -66,7 +66,7 @@ cmd=(
--obtype "${OBTYPE}"
--obs_day "${YYMMDD}"
)
echo "CALLING: ${cmd[*]}"
printf "CALLING: ${cmd[*]}"
"${cmd[@]}" || print_err_msg_exit "Error calling get_obs.py"
#
#-----------------------------------------------------------------------
Expand All @@ -77,6 +77,7 @@ echo "CALLING: ${cmd[*]}"
#-----------------------------------------------------------------------
#
mkdir -p ${WFLOW_FLAG_FILES_DIR}
file_bn="get_obs_$(echo_lowercase ${OBTYPE})"
# ${VARNAME,,} converts contents of VARNAME to lowercase
file_bn="get_obs_${OBTYPE,,}"
touch "${WFLOW_FLAG_FILES_DIR}/${file_bn}_${YYMMDD}_complete.txt"

10 changes: 5 additions & 5 deletions jobs/GRIDSTAT_OR_POINTSTAT.sh
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ sections=(
for sect in ${sections[*]} ; do
source_yaml ${GLOBAL_VAR_DEFNS_FP} ${sect}
done
# Sets up PYTHONPATH, VERBOSE and ACCUM_ARG environment variables
# Sets up PYTHONPATH, ACCUM_ARG, ENSMEM_ARG and VERBOSE environment variables
. $USHdir/set_job_env.sh
#
#-----------------------------------------------------------------------
Expand All @@ -50,23 +50,23 @@ scrfunc_fp=$( $READLINK -f "${BASH_SOURCE[0]}" )
scrfunc_fn=$( basename "${scrfunc_fp}" )
scrfunc_dir=$( dirname "${scrfunc_fp}" )

print_info_msg "
printf "
========================================================================
Entering script: \"${scrfunc_fn}\"
In directory: \"${scrfunc_dir}\"
========================================================================"
#
# Call the run script
#
python $SCRIPTSdir/gridstat_or_pointstat.py ${VERBOSE_FLAG} ${ACCUM_ARG} \
python $SCRIPTSdir/gridstat_or_pointstat.py ${VERBOSE_FLAG} \
--config="${GLOBAL_VAR_DEFNS_FP}" \
--cycle_date="${YYMMDD}${HH}" \
--ensmem_index="${ENSMEM_INDX}" \
--field_group="${FIELD_GROUP}" \
--fcst_level="${FCST_LEVEL}" \
--fcst_thresh="${FCST_THRESH}" \
--obtype="${OBTYPE}" \
--obs_dir="${OBS_DIR}" || \
--obs_dir="${OBS_DIR}" \
${ACCUM_ARG} ${ENSMEM_ARG} || \
print_err_msg_exit "\
Call to \"gridstat_or_pointstat.py\" from \"${scrfunc_fn}\" failed."

17 changes: 14 additions & 3 deletions jobs/GRIDSTAT_OR_POINTSTAT_ENSMEAN.sh
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,8 @@ sections=(
for sect in ${sections[*]} ; do
source_yaml ${GLOBAL_VAR_DEFNS_FP} ${sect}
done
# Sets up PYTHONPATH and VERBOSE environment variables
. $USHdir/set_job_env.sh
#
#-----------------------------------------------------------------------
#
Expand All @@ -49,14 +51,23 @@ scrfunc_fp=$( $READLINK -f "${BASH_SOURCE[0]}" )
scrfunc_fn=$( basename "${scrfunc_fp}" )
scrfunc_dir=$( dirname "${scrfunc_fp}" )

print_info_msg "
printf "
========================================================================
Entering script: \"${scrfunc_fn}\"
In directory: \"${scrfunc_dir}\"
========================================================================"
#
# Call the run script
#
$SCRIPTSdir/gridstat_or_pointstat_ensmean.sh || \
python $SCRIPTSdir/gridstat_or_pointstat_ensmean.py ${VERBOSE_FLAG} \
--config="${GLOBAL_VAR_DEFNS_FP}" \
--cycle_date="${YYMMDD}${HH}" \
--field_group="${FIELD_GROUP}" \
--obs_dir="${OBS_DIR}" \
--obtype="${OBTYPE}" \
--fcst_level="${FCST_LEVEL}" \
--fcst_thresh="${FCST_THRESH}" \
${ACCUM_ARG} || \
print_err_msg_exit "\
Call to \"gridstat_or_pointstat_ensmean.sh\" from \"${scrfunc_fn}\" failed."
Call to \"gridstat_or_pointstat_ensmean.py\" from \"${scrfunc_fn}\" failed."

17 changes: 13 additions & 4 deletions jobs/GRIDSTAT_OR_POINTSTAT_ENSPROB.sh
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,8 @@ sections=(
for sect in ${sections[*]} ; do
source_yaml ${GLOBAL_VAR_DEFNS_FP} ${sect}
done
# Sets up PYTHONPATH and VERBOSE environment variables
. $USHdir/set_job_env.sh
#
#-----------------------------------------------------------------------
#
Expand All @@ -49,15 +51,22 @@ scrfunc_fp=$( $READLINK -f "${BASH_SOURCE[0]}" )
scrfunc_fn=$( basename "${scrfunc_fp}" )
scrfunc_dir=$( dirname "${scrfunc_fp}" )

print_info_msg "
printf "
========================================================================
Entering script: \"${scrfunc_fn}\"
In directory: \"${scrfunc_dir}\"
========================================================================"
#
# Call the run script
#
$SCRIPTSdir/gridstat_or_pointstat_ensprob.sh || \
python $SCRIPTSdir/gridstat_or_pointstat_ensprob.py ${VERBOSE_FLAG} \
--config="${GLOBAL_VAR_DEFNS_FP}" \
--cycle_date="${YYMMDD}${HH}" \
--field_group="${FIELD_GROUP}" \
--obs_dir="${OBS_DIR}" \
--obtype="${OBTYPE}" \
--fcst_level="${FCST_LEVEL}" \
--fcst_thresh="${FCST_THRESH}" \
${ACCUM_ARG} || \
print_err_msg_exit "\
Call to \"gridstat_or_pointstat_ensprob.sh\" from \"${scrfunc_fn}\" failed."

Call to \"gridstat_or_pointstat_ensprob.py\" from \"${scrfunc_fn}\" failed."
2 changes: 1 addition & 1 deletion jobs/MODE.sh
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ scrfunc_fp=$( $READLINK -f "${BASH_SOURCE[0]}" )
scrfunc_fn=$( basename "${scrfunc_fp}" )
scrfunc_dir=$( dirname "${scrfunc_fp}" )

print_info_msg "
printf "
========================================================================
Entering script: \"${scrfunc_fn}\"
In directory: \"${scrfunc_dir}\"
Expand Down
4 changes: 2 additions & 2 deletions jobs/PB2NC_OBS.sh
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ scrfunc_fp=$( $READLINK -f "${BASH_SOURCE[0]}" )
scrfunc_fn=$( basename "${scrfunc_fp}" )
scrfunc_dir=$( dirname "${scrfunc_fp}" )

print_info_msg "
printf "
========================================================================
Entering script: \"${scrfunc_fn}\"
In directory: \"${scrfunc_dir}\"
Expand All @@ -64,5 +64,5 @@ python $SCRIPTSdir/pb2nc_obs.py ${VERBOSE_FLAG} \
--cycle_date="${YYMMDD}${HH}" \
--obtype="${OBTYPE}" || \
print_err_msg_exit "\
Call to \"pb2nc_obs.sh\" from \"${scrfunc_fn}\" failed."
Call to \"pb2nc_obs.py\" from \"${scrfunc_fn}\" failed."

17 changes: 14 additions & 3 deletions jobs/PCPCOMBINE.sh
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,8 @@ sections=(
for sect in ${sections[*]} ; do
source_yaml ${GLOBAL_VAR_DEFNS_FP} ${sect}
done
# Sets up PYTHONPATH, ACCUM_ARG, ENSMEM_ARG, and VERBOSE environment variables
. $USHdir/set_job_env.sh

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In addition to PYTHONPATH, any flags for a given run script that are optional (i.e. the environment variable might not be set at runtime) are set in this step. See ush/set_job_env.sh for details.

#
#-----------------------------------------------------------------------
#
Expand All @@ -50,15 +52,24 @@ scrfunc_fp=$( $READLINK -f "${BASH_SOURCE[0]}" )
scrfunc_fn=$( basename "${scrfunc_fp}" )
scrfunc_dir=$( dirname "${scrfunc_fp}" )

print_info_msg "
printf "
========================================================================
Entering script: \"${scrfunc_fn}\"
In directory: \"${scrfunc_dir}\"
========================================================================"
#
# Call the run script
#
$SCRIPTSdir/pcpcombine.sh || \
python $SCRIPTSdir/pcpcombine.py ${VERBOSE_FLAG} \
--config="${GLOBAL_VAR_DEFNS_FP}" \
--cycle_date="${YYMMDD}${HH}" \
--field_group="${FIELD_GROUP}" \
--fcst_or_obs="${FCST_OR_OBS}" \
--obs_dir="${OBS_DIR}" \
--obtype="${OBTYPE}" \
--fcst_level="${FCST_LEVEL}" \
--fcst_thresh="${FCST_THRESH}" \
${ACCUM_ARG} ${ENSMEM_ARG} || \
print_err_msg_exit "\
Call to \"pcpcombine.sh\" from \"${scrfunc_fn}\" failed."
Call to \"pcpcombine.py\" from \"${scrfunc_fn}\" failed."

2 changes: 1 addition & 1 deletion jobs/POINT2GRID.sh
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ scrfunc_fp=$( $READLINK -f "${BASH_SOURCE[0]}" )
scrfunc_fn=$( basename "${scrfunc_fp}" )
scrfunc_dir=$( dirname "${scrfunc_fp}" )

print_info_msg "
printf "
========================================================================
Entering script: \"${scrfunc_fn}\"
In directory: \"${scrfunc_dir}\"
Expand Down
2 changes: 1 addition & 1 deletion jobs/REGRIDDATAPLANE.sh
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ scrfunc_fp=$( $READLINK -f "${BASH_SOURCE[0]}" )
scrfunc_fn=$( basename "${scrfunc_fp}" )
scrfunc_dir=$( dirname "${scrfunc_fp}" )

print_info_msg "
printf "
========================================================================
Entering script: \"${scrfunc_fn}\"
In directory: \"${scrfunc_dir}\"
Expand Down
2 changes: 1 addition & 1 deletion jobs/TCPAIRS.sh
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ scrfunc_fp=$( $READLINK -f "${BASH_SOURCE[0]}" )
scrfunc_fn=$( basename "${scrfunc_fp}" )
scrfunc_dir=$( dirname "${scrfunc_fp}" )

print_info_msg "
printf "
========================================================================
Entering script: \"${scrfunc_fn}\"
In directory: \"${scrfunc_dir}\"
Expand Down
2 changes: 1 addition & 1 deletion jobs/TCRMW.sh
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ scrfunc_fp=$( $READLINK -f "${BASH_SOURCE[0]}" )
scrfunc_fn=$( basename "${scrfunc_fp}" )
scrfunc_dir=$( dirname "${scrfunc_fp}" )

print_info_msg "
printf "
========================================================================
Entering script: \"${scrfunc_fn}\"
In directory: \"${scrfunc_dir}\"
Expand Down
2 changes: 1 addition & 1 deletion jobs/TCSTAT.sh
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ scrfunc_fp=$( $READLINK -f "${BASH_SOURCE[0]}" )
scrfunc_fn=$( basename "${scrfunc_fp}" )
scrfunc_dir=$( dirname "${scrfunc_fp}" )

print_info_msg "
printf "
========================================================================
Entering script: \"${scrfunc_fn}\"
In directory: \"${scrfunc_dir}\"
Expand Down
2 changes: 1 addition & 1 deletion parm/metplus/EnsembleStat.conf
Original file line number Diff line number Diff line change
Expand Up @@ -221,7 +221,7 @@ OBS_{{METPLUS_TOOL_NAME}}_WINDOW_END = {OBS_WINDOW_END}
{#-
Import the file containing jinja macros.
#}
{%- import metplus_templates_dir ~ '/metplus_macros.jinja' as metplus_macros %}
{%- import 'metplus_macros.jinja' as metplus_macros %}

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For some reason the old syntax doesn't work in the python context. This has no change in functionality.


{#-
Jinja requires certain variables to be defined globally within the template
Expand Down
2 changes: 1 addition & 1 deletion parm/metplus/GenEnsProd.conf
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,7 @@ STAGING_DIR = {{staging_dir}}
{#-
Import the file containing jinja macros.
#}
{%- import metplus_templates_dir ~ '/metplus_macros.jinja' as metplus_macros %}
{%- import 'metplus_macros.jinja' as metplus_macros %}

{#-
Jinja requires certain variables to be defined globally within the template
Expand Down
2 changes: 1 addition & 1 deletion parm/metplus/GridStat_ensmean.conf
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ OBTYPE = {{obtype}}
{#-
Import the file containing jinja macros.
#}
{%- import metplus_templates_dir ~ '/metplus_macros.jinja' as metplus_macros %}
{%- import 'metplus_macros.jinja' as metplus_macros %}

{#-
Set the probabilistic threshold to be used for the forecast field. If
Expand Down
2 changes: 1 addition & 1 deletion parm/metplus/GridStat_ensprob.conf
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,7 @@ OBTYPE = {{obtype}}
{#-
Import the file containing jinja macros.
#}
{%- import metplus_templates_dir ~ '/metplus_macros.jinja' as metplus_macros %}
{%- import 'metplus_macros.jinja' as metplus_macros %}

{#-
Set the probabilistic threshold to be used for the forecast field. If
Expand Down
2 changes: 1 addition & 1 deletion parm/metplus/PointStat_ensmean.conf
Original file line number Diff line number Diff line change
Expand Up @@ -161,7 +161,7 @@ OBTYPE = {{obtype}}
{#-
Import the file containing jinja macros.
#}
{%- import metplus_templates_dir ~ '/metplus_macros.jinja' as metplus_macros %}
{%- import 'metplus_macros.jinja' as metplus_macros %}

{#-
Set the probabilistic threshold to be used for the forecast field. If
Expand Down
2 changes: 1 addition & 1 deletion parm/metplus/PointStat_ensprob.conf
Original file line number Diff line number Diff line change
Expand Up @@ -163,7 +163,7 @@ OBTYPE = {{obtype}}
{#-
Import the file containing jinja macros.
#}
{%- import metplus_templates_dir ~ '/metplus_macros.jinja' as metplus_macros %}
{%- import 'metplus_macros.jinja' as metplus_macros %}

{#-
Set the probabilistic threshold to be used for the forecast field. If
Expand Down
Loading
Loading