Planning the reorganization of the config.yaml to make nearly everything configurable and greatly simplify METplus conf file logic

[mkavulich]

Description

Once #33 is merged, we will have much more flexibility in how we store data in the config file. Using bash scripts restricts us to reading a flat dictionary of variables, while python run scripts allow us to read multi-level dictionaries. We can use this to solve several open issues by introducing new sections and subsections to the config.yaml file

#35, allowing multiple obs types for a single field

To solve #35, we can introduce a new top-level config section

• obs: will have a subsection for each observation type
  • This will allow us to migrate a lot of the clunky obs-specific variables out of the top-level verification: section. For example verification: CCPA_OBS_AVAIL_INTVL_HRS could now be obs:CCPA:file_interval, verification:CCPA_OBS_DIR could now be obs:CCPA:staging_dir or something similar
  • The OBS_*_FN_TEMPLATES variables could now be a proper dictionary in each instead of a hacky list

Here is an example of what this could look like:

obs:
  CCPA:
    file_interval: 3600 # This could be specified in seconds rather than hours, to allow future sub-hourly verification capabilities
    data_store: hpss # Instead of `verification: OBS_DATA_STORE_CCPA: hpss`
    APCP:
      fn_template: '{valid?fmt=%Y%m%d}/ccpa.t{valid?fmt=%H}z.{{ "%02d" % verification.CCPA_OBS_AVAIL_INTVL_HRS / 3600 }}h.hrap.conus.gb2'
  MRMS:
    file_interval: 3600
    irregular_times: True # This setting would invoke calls to `select_validtime_obs()` to select the appropriate observation file for obs like MRMS that are at irregular times. Currently this is hard-coded for certain ob types in `scripts/get_obs.py`
    REFC: 
      fn_template: '{valid?fmt=%Y%m%d}/MergedReflectivityQCComposite_00.50_{valid?fmt=%Y%m%d}-{valid?fmt=%H%M%S}.grib2'
    RETOP: 
      fn_template: '{valid?fmt=%Y%m%d}/EchoTop_18_00.50_{valid?fmt=%Y%m%d}-{valid?fmt=%H%M%S}.grib2'

#17, allow specification of threshold data in the config file

To solve #17, we can introduce another new top-level config section fields:, with a subsection for each field type

• we can move all the information currently found in [parm/metplus/vx_configs](https://github.com/dtcenter/dtc-vx-workflow/tree/develop/parm/metplus/vx_configs) into this new config section

Here is an example of what this could look like:

Current:

ASNOW:
    ASNOW:
        A6: ['gt0.0', 'ge2.54', 'ge5.08', 'ge10.16', 'ge20.32']
        A24: ['gt0.0', 'ge2.54', 'ge10.16', 'ge20.32', 'ge30.48']
REFC:
    REFC%%MergedReflectivityQCComposite:
        L0%%Z500: ['ge20', 'ge30', 'ge40', 'ge50']
RETOP:
    RETOP%%EchoTop18:
        L0%%Z500: ['ge20', 'ge30', 'ge40', 'ge50']
UPA:
    TMP:
        P1000: &adpupa_tmp_threshes
               []
        P925: *adpupa_tmp_threshes
        P850: *adpupa_tmp_threshes
        P700: *adpupa_tmp_threshes
        P500: *adpupa_tmp_threshes
...
SFC:
  CRAIN%%PRWE:
    L0%%Z0:
      thresholds: ['ge1.0%%ge161&&le163']

Proposed new:

fields:
  ASNOW:
    ASNOW:
      A6:
        thresholds: ['gt0.0', 'ge2.54', 'ge5.08', 'ge10.16', 'ge20.32']
      A24:
        thresholds: ['gt0.0', 'ge2.54', 'ge10.16', 'ge20.32', 'ge30.48']
  RADAR:
    REFC:
      obs_field_name: MergedReflectivityQCComposite
      L0:
        obs_level_name: Z500
        thresholds: ['ge20', 'ge30', 'ge40', 'ge50']
    RETOP:
      obs_field_name: EchoTop18
      L0:
        obs_level_name: Z500
        thresholds: ['ge20', 'ge30', 'ge40', 'ge50']
  UPA:
    TMP:
      P1000: &adpupa_tmp_threshes
        thresholds: []
      P925: *adpupa_tmp_threshes
      P850: *adpupa_tmp_threshes
      P700: *adpupa_tmp_threshes
      P500: *adpupa_tmp_threshes
...
  SFC:
    CRAIN:
      obs_field_name: PRWE
      L0:
        obs_level_name: Z0
        obs_thresholds: ['ge161&&le163']
        thresholds: ['ge1.0']

Note: in the current dictionary there is some complex string-splitting logic to associate the observation field name with the forecast field name. This requires a lot of unnecessary and complex python logic that we can get rid of by just rearranging into the above dictionary structure. While it's somewhat arbitrary which level of the dictionary specifies the name (observation files vs forecast files), it is important that there is a direct one-to-one association. Deciding to use the observation type name at the top-level key and using "fcst_field_name:" as a key underneath that level means that in the future we may be able to specify different observation type names for different ensemble forecast files (see #20). It turns out this does not work: because the observation name can be the same for different model fields, we need to use the forecast field as the dictionary key to avoid duplicate keys (which are disallowed)

If not specified, fields such as obs_field_name, obs_level_name, etc. would default to the same as the forecast value.

The way I envision this working is:

• we will only have one set of default thresholds, taken from vx_config_det.yaml. We can give examples of different thresholds that are currently stored in the other files in WE2E tests where appropriate
• This process will deprecate the variables VX_CONFIG_DET_FN, VX_CONFIG_ENS_FN

The only downside of this approach I can see is that we can no longer have different thresholds for verifying the individual members vs the whole ensemble. Is that functionality we envision needing?

Code we can remove

• ush/set_vx_params.py Any information that is coming from this file can instead be assigned in one of these new config sections
• parm/metplus/metplus_macros.jinja: can be replaced with logic in make_var_lists; this makes for easier to understand variable construction

Other improvements

• Break out task-specific config settings into task-specific sections; allow nested sub-sections for better organization #9 can be solved (if it hasn't been already).
• We can pull out a ton of conditional logic currently in the conf files, and make them simple definitions in this configuration dictionary.
• Some other clunky obs/field specific variable names can be simplified as well. For example:
  • verification: OBS_CCPA_APCP_FN_TEMPLATE_PCPCOMBINE_OUTPUT: 'xyzabc' can become `pcpcombine: output_fn: APCP: 'xyzabc'
  • verification: AIRNOW_INPUT_FORMAT: "airnowhourlyaqobs" can become field: AIRNOW: format: "airnowhourlyaqobs"

Examples

Here are a few examples of how our current config files would translate to this new format: (will work on this later)

[mkavulich]

After more work, I have decided that there is no way to make this raw dictionary solution work, since we have multiple entries that have either identical forecast field names (e.g. CAPE) or observation fields names (e.g. PRWE). So with Claude's help I have moved to a dictionary of lists.

Current:

ASNOW:
    ASNOW:
        A6: ['gt0.0', 'ge2.54', 'ge5.08', 'ge10.16', 'ge20.32']
        A24: ['gt0.0', 'ge2.54', 'ge10.16', 'ge20.32', 'ge30.48']
REFC:
    REFC%%MergedReflectivityQCComposite:
        L0%%Z500: ['ge20', 'ge30', 'ge40', 'ge50']
RETOP:
    RETOP%%EchoTop18:
        L0%%Z500: ['ge20', 'ge30', 'ge40', 'ge50']
UPA:
    TMP:
        P1000: &adpupa_tmp_threshes
               []
        P925: *adpupa_tmp_threshes
        P850: *adpupa_tmp_threshes
        P700: *adpupa_tmp_threshes
        P500: *adpupa_tmp_threshes
...
SFC:
  CRAIN%%PRWE:
    L0%%Z0:
      thresholds: ['ge1.0%%ge161&&le163']

Proposed new:

fields:
  ASNOW:
    - fcst_name: ASNOW_06
      fcst_levels: [A6]
      fcst_thresholds: ['gt0.0', 'ge2.54', 'ge5.08', 'ge10.16', 'ge20.32']
      fcst_options: 'set_attr_lead = "{lead?fmt=%H%M%S}";'
    - fcst_name: ASNOW_24
      fcst_levels: [A24]
      fcst_thresholds: ['gt0.0', 'ge2.54', 'ge10.16', 'ge20.32', 'ge30.48']
      fcst_options: 'set_attr_lead = "{lead?fmt=%H%M%S}";'
  REFC:
    - fcst_name: REFC
      obs_name: MergedReflectivityQCComposite
      fcst_levels: [L0]
      obs_levels: [Z500]
      fcst_thresholds: ['ge20', 'ge30', 'ge40', 'ge50']
      fcst_options: 'set_attr_lead = "{lead?fmt=%H%M%S}"; cnt_thresh = [ >15 ]; cnt_logic = UNION;'
      obs_options: 'censor_thresh = [eq-999, <-20]; censor_val = [-9999, -20]; cnt_thresh = [ >15 ]; cnt_logic = UNION;'
  RETOP:
    - fcst_name: RETOP
      obs_name: EchoTop18
      fcst_levels: [L0]
      obs_levels: [Z500]
      fcst_thresholds: ['ge20', 'ge30', 'ge40', 'ge50']
      fcst_options: 'set_attr_lead = "{lead?fmt=%H%M%S}"; convert(x) = x * 3.28084 * 0.001; cnt_thresh = [ >0 ]; cnt_logic = UNION;'
      obs_options: 'convert(x) = x * 3280.84 * 0.001; censor_thresh = [<=-9.84252,eq-3.28084]; censor_val = [-9999,-16.4042]; cnt_thresh = [ >0 ]; cnt_logic = UNION;'

...
  SFC:
    - fcst_name: CRAIN
      obs_name: PRWE
      fcst_levels: [L0]
      obs_levels: [Z0]
      fcst_thresholds: ['ge1.0']
      obs_thresholds: ['ge161&&le163']
      fcst_options: 'set_attr_lead = "{lead?fmt=%H%M%S}";'

Note that there are a bunch of new "options" entries in here; I have moved a whole bunch of logic hard-coded in the conf files to also be specifiable in the config file like this.

This change will result in further benefits:

• We no longer need to have separate GridStat tasks for different accumulation hours for APCP and ASNOW.
• We can also combine REFC and RETOP into a RADAR field group
• We no longer need a bunch of hard-coded OPTIONS settings in conf files. These are also stored in this fields: dictionary

I have also decided not to lose the capability to have different thresholds for ensemble vs deterministic. There is now support for including fields: in both the ensemble section (for all ensemble tasks) as well as task-specific sections.