Ability to choose between multiple observations in one file

[mvertens]

Summary

Some variables can be compared against more than one observational dataset — for example ERA5 at native resolution or ERA5 regridded to 1°, or ERAI or ERA5. Until now, supporting both meant keeping two near-identical copies of the whole defaults file (adf_variable_defaults.yaml and adf_variable_defaults_era5-1deg.yaml), which was confusing and painful to keep in sync.

This change lets a variable list several observations in one place, and lets you pick which one a run uses with a single config setting — so the duplicate file goes away entirely.

How you use it

In the variable defaults, a variable lists its observation options under
obs_datasets (each option is self-contained — its own file, name, units, and derivation):

TREFHT:
  category: "Surface variables"
  new_unit: "K"
  obs_datasets:
    - obs_name: "ERA5"
      obs_file: "TREFHT_ERA5_monthly_climo_197901-202112.nc"
    - obs_name: "ERA5_1deg"
      obs_file: "TREFHT_ERA5_monthly_climo_1degree_197901-202112.nc"

Then, in diag_basic_info, one setting chooses which observation the whole run uses:

diag_basic_info:
    obs_source: "ERA5_1deg"      # match an obs_name; omit to use each variable's default

What gets picked:

obs_source is matched against each variable's obs_name. The rules:

Situation	Which observation is used
obs_source matches one of the variable's obs_names	that one
obs_source is set but the variable doesn't offer it	the first one listed (its default) + a debug \note
obs_source is not set at all	the first one listed (its default)
the variable has just a single observation	that one, always
the variable has no observation	skipped

So the convention is simple: list the default observation first, and each run picks a source with one line. A variable that doesn't offer that source quietly falls back to its default, so a run never loses a comparison unexpectedly.

A couple of nice consequences:

• One file instead of two. All the 1° ERA5 observations were merged in as second entries, and the duplicate adf_variable_defaults_era5-1deg.yaml was deleted. To switch resolutions you now flip one config line instead of swapping whole files.
• Each observation is self-contained. An option carries its own file, variable name, unit scaling, and derivation formula — so two sources for the same variable can differ (e.g. ERA5 vs ERAI, or a field that needs a different unit conversion
  at 1°).
• This does not effect existing runs. A variable with a single observation behaves exactly as before, and if obs_source isn't set everything uses its default. The model side is untouched.

Testing

Verified end-to-end: the real converted YAML resolves correctly through the obs catalog and the data loader, obs_source selects native vs 1° (and handles the ERAI-vs-ERA5 case) as expected, and single-observation entries are unchanged. All
three defaults files were converted with an automated check confirming no observation value or other attribute was altered.