Bug-fix extraction from generation-cleanout-v2 - #87
Merged
Conversation
(cherry picked from commit fc591ca)
…xog='none') The EOF-weighting fix (PR #74) corrected the *monthly* global tas variance, but the *annual/decadal* global variability was still ~2.5x too small: the generated noise was temporally white/anti-persistent (lag-1 autocorr ~-0.1) whereas real global-mean tas is strongly red (lag-1 ~+0.57). Root cause is the VAR-X exogenous regressor, not the EOF basis or the seasonal t_glob removal. Stage-by-stage (CanESM5 tas, validated against piControl): target (piControl) annual std 0.089 K lag-1 +0.57 training anomaly global mean 0.081 K +0.56 (red) in-sample retained PCs -> global 0.074 K +0.51 (red) VAR-X generated, use_exog='all' 0.038 K +0.18 (white) VAR-X generated, use_exog='none' 0.076 K +0.53 (red) Using the smoothed global temperature (t_glob) as a VAR-X exogenous regressor absorbs the persistent low-frequency global variability into the deterministic forced term. At generation t_glob is the prescribed (smooth, internally invariant) trajectory, so that power is never regenerated -> white noise. With use_exog='none' the persistence stays in the AR dynamics and is reproduced (CanESM5 86% of target annual std, ACCESS-ESM1-5 79%; monthly variance unchanged). The temperature-dependent mean/seasonal response is already captured by the seasonal model, so the exog regressor is redundant. Changes: - MeteorNoiseGenerator and train_*_from_cmip6 default use_exog 'temp_only'->'none' - interface _get_default_config: tas 'all'->'none' (pr was already 'none') - 'temp_only'/'all' retained for backward compatibility (documented as suppressing low-frequency global variability; 'temp_only' is also unstable at high lag_order) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> (cherry picked from commit 75759b3)
(cherry picked from commit 4f5df80)
(cherry picked from commit 25f0686)
(cherry picked from commit e16947b)
(cherry picked from commit 402e20b)
The Gamma transform was conflating the seasonal cycle and the climate-change
trend with internal variability:
- Time-series path: the Gaussian was fit on the whole flattened
(n_real, n_time) ensemble, so sigma was dominated by the wet/dry-season
swing and the inter-realization noise band got squashed in the CDF→PPF
roundtrip.
- Gridded path: the transform was applied per year-slice, so the per-
gridpoint Gaussian mean was the local 1-year climatology, which
normalized every year to its own local distribution and wiped the
climate-change trend at each gridpoint.
Fit and apply per month-of-year (12 fits) so the seasonal cycle lives in
the per-month means instead of sigma, and on the gridded side build a
single full-window transformed ensemble up front so the annual / monthly /
climatology slicers just read from it. Old pooled fit/apply functions are
kept for backwards compatibility; the seasonal variants are wired through
VariableTransformConfig and only used when present.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
(cherry picked from commit 706a94a)
(cherry picked from commit a11f269)
maritsandstad
approved these changes
Jun 30, 2026
maritsandstad
left a comment
maritsandstad
left a comment
Collaborator
There was a problem hiding this comment.
I am fine with this, minor comments which could be discussed and no big deal if we defer them to later
- Remove the leftover `print(n_lat, n_lon, n_time)` debug call in noise_generator._generate_realizations_unified_implementation. - Expand the one-line geo_data_utils module docstring into a grouped inventory of the helpers it exposes.
Owner
Author
|
Thanks @maritsandstad — pushed
For the third comment (folding most of |
benmsanderson
marked this pull request as ready for review
June 30, 2026 19:17
maritsandstad
approved these changes
Jun 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Extracts the bug fixes that had accumulated on
generation-cleanout-v2into a focused PR againstbase, leaving the FastMIP operational scripts and notebooks behind.The 10 commits here are cherry-picks of:
9e76742,b90204censure_pathfor netcdf dump, plus various scaling-timeseries / diverse-input fixes5fb128euse_exogto'none'585614a,16ad7179250e37b3e1b75,a5c5656,8007a21,7c8e83ainterface field generation fix on base(introducesGenerationInputs)The cherry-picks preserved Marit's and Ben's authorship on each commit.
What was deliberately left out
scripts/FastMIP_*(4 scripts)notebooks/FASTMIP_*.ipynb,notebooks/FastMIP_*.ipynb,notebooks/aggregate_to_regions.py,notebooks/FAIR_GSAT/data/FASTMIP_phase1/FAIR_GSAT/...FastMIP_README.md,FastMIP_env_setup.sh,conda_env_fastmip.ymlnotebooks/GCAM_predict.ipynbre-execution bundled into the linting commitDiff
10 files, +1460 / −260 — all under
src/,tests/, andCHANGELOG.rst. No notebooks, no operational scripts.Test plan
test_meteor_interface.py,test_geo_data_utils.py,test_noise_generator.py)black --check,ruff check,isort --check-onlyall clean