account for temporal frequency in source data

[siligam]

temporal frequency of source must be finer than matching table frequency for Pymor to include it in the cmorization process.

Lets say temporal frequency of source files is quarterly. The variable matches Omon and Oyr tables. In this case, Pymor must drop processing data for monthly table as source temporal frequency is coarser than monthly frequency.

However, It should process data for yearly table.

In the current implementation this is not the case.

[pgierz]

Can you produce a failing unit test for this? I can take care of correctly implementing it if I have a concrete example...

[siligam]

An example to reproduce this issue is in one of the exercises in pymor_workshop esm-tools/pycmor_workshop#43

Source data frequency is Quarterly:

cdo -s showtimestamp ../data/qCO2f_fesom_quaterly_3001.nc
cdi  warning (cdfScanVarAttr): NetCDF: Variable not found - time_bnds
  3001-01-01T00:00:00  3001-04-01T00:00:00  3001-07-01T00:00:00  3001-10-01T00:00:00

If this variable is matching, Omon and Oyr table, the cmorizer.CMORizer includes them in the pipeline. It does not consider the frequency of source data as _check_is_subperiod function is turned off in the validation function https://github.com/esm-tools/pymor/blob/5fe78529b467a3ad0e924f8f277e4192868a5bf7/src/pymor/core/cmorizer.py#L439-L446

With the current setting, Pymor produces cmorized monthly data from the quarterly data which is not desirable.

[mandresm]

@pgierz mentioned in a Pymor meeting that we should cache metadata of files to then run checks on this cache. @siligam has already put work on these. This can be used ask for the frequencies of all files to be processed and check whether those freq. are larger than the requested in tables.

[pgierz]

@siligam is this related to #186 and/or #188?

[siligam]

#186 or #188 does not solve this issue entirely. they provide the necessary logic to determine the frequency check. Notice _check_is_subperiod function is commented out (see above). I think this function needs to be updated with the new logic and then re-enable in the validation checks so the framework at-least raises an error if it does not meet the criteria.

[pgierz]

The open question here is how to embed the new frequency analysis tools in the pipeline.

• Example: Yearly SST is requested. User has data that is typically on monthly output. They change to daily for some period, and then back to monthly. Our tool should be able to nevertheless produce the required output (averaged) output frequency, e.g. OYear. Users should be explicitly informed that certain sub-periods needed to be averaged.

New work from @siligam shows that we have a much better semantic understanding of how the timestamps in the data now look, so we can do more intelligent error message

• Blocked by Fix frequency inference for monthly data with missing months #199 (TODO @pgierz)

Several Options for including this
1. Framework level [@mandresm, @pgierz]. As a pymor.ensure_temporal_consistency = True switch. This will also have implications for any function that attempts to manipulate the time axis.
2. As one of the processing steps, see below:

rules:
   - name: Woohoo My Rule
     cmor_name: "tos"
     model_name: "sst"
     pipelines:  # Default behaviour is currently ["default"], could be changed to ["run_checks", "default"]
         - run_checks
         - default
         - fancy_custom
pipelines:
    - run_checks:
         - check_temporal_consistency

Directly in Python:

def check_temporal_consistency(data, rule):
    if data.is_temporaly_conform(rule):  # This is @siligam accessor
        raise UserError("You have inconsistent temporal data, are you sure about this?")
    return data