`catalogbuilder` task failing after package update

[ilaflott]

behavior has changed enough that the catalog builder's assumptions/behavior no longer match the workflow requirements for the tool

[ilaflott]

thanks to @singhd789 's sleuthing we have an error message! pretty much on point...

[WARNING:      gen_intake_gfdl.py:      create_catalog] !!!!! IMPORTANT: RECENT CHANGES TO THE CATALOG BUILDER MAY AFFECT EXISTING WORKFLOWS !!!!!
Traceback (most recent call last):
  File "/app/cylc-flow-tools/lib/python3.11/site-packages/catalogbuilder/intakebuilder/getinfo.py", line 166, in getInfoFromGFDLDRS
    dictInfo[input_path_template[i]] = stemdir[(j)]
                                       ~~~~~~~^^^^^
IndexError: list index out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/cylc-flow-tools/lib/python3.11/site-packages/catalogbuilder/intakebuilder/getinfo.py", line 168, in getInfoFromGFDLDRS
    raise IndexError("Check configuration. Is input path template set correctly?")
IndexError: Check configuration. Is input path template set correctly?

[Ciheim]

Hmm.. I'd guess this is a result of a few configuration keys changing: output_path_template -> input_path_template, output_file_template -> input_file_template @ilaflott @singhd789

[ilaflott]

So if i'm understanding correctly, we basically need to switch the input/output arguments?

On the fre-workflows side, this is how the catalogbuilder is being invoked:

fre-workflows/bin/data-catalog

Line 25 in 106de2e

fre catalog build $* $PP_DIR $PP_DIR/catalog

Within fre-cli, the CLI is defined here: https://github.com/NOAA-GFDL/fre-cli/blob/4ec5686e6a774768fd9fbc5a6b187fb8db9e6ac9/fre/catalog/frecatalog.py#L18-L41

@catalog_cli.command()
#TODO arguments dont have help message. So consider changing arguments to options?
@click.argument('input_path', required = False, nargs = 1)
#, help = 'The directory path with the datasets to be cataloged. E.g a GFDL PP path till /pp')
@click.argument('output_path', required = False, nargs = 1)
#, help = 'Specify output filename suffix only. e.g. catalog')
@click.option('--config', required = False, type = click.Path(exists = True), nargs = 1,
              help = 'Path to your yaml config, Use the config_template in intakebuilder repo')
@click.option('--filter_realm',  nargs = 1)
@click.option('--filter_freq',  nargs = 1)
@click.option('--filter_chunk',  nargs = 1)
@click.option('--verbose', is_flag = True, default = False)
@click.option('--overwrite', is_flag = True, default = False)
@click.option('--append', is_flag = True, default = False)
@click.option('--slow', is_flag = True, default = False,
    help = "Open NetCDF files to retrieve additional vocabulary (standard_name and intrafile static variables")
@click.option('--strict', is_flag = True, default = False,
    help = "Ensure output catalog is strictly compliant with schema")
@click.pass_context
def build(context, input_path = None, output_path = None, config = None, filter_realm = None,
          filter_freq = None, filter_chunk = None, verbose = False, overwrite = False, append = False, slow = False, strict = False):
    # pylint: disable=unused-argument
    """ - Generate .csv and .json files for catalog """
    context.forward(gen_intake_gfdl.create_catalog_cli)

and we have this fre catalog build --help output:

(fre-cli) ldt-4788479: fre-cli $] fre catalog build --help
Usage: fre catalog build [OPTIONS] [INPUT_PATH] [OUTPUT_PATH]

  - Generate .csv and .json files for catalog

Options:
  --config PATH        Path to your yaml config, Use the config_template in
                       intakebuilder repo
  --filter_realm TEXT
  --filter_freq TEXT
  --filter_chunk TEXT
  --verbose
  --overwrite
  --append
  --slow               Open NetCDF files to retrieve additional vocabulary
                       (standard_name and intrafile static variables
  --strict             Ensure output catalog is strictly compliant with schema
  --help               Show this message and exit.