Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
9759f42
working on FASTMIP phase 1 as example for timeseries scaling
mauradewey Mar 11, 2026
5ee7683
splitting notebooks for phase 1 and phase 2
mauradewey Mar 11, 2026
2f9b83a
Merge remote-tracking branch 'origin/base' into fastmip
mauradewey Mar 11, 2026
ba2bab0
Merge remote-tracking branch 'origin/base' into fastmip
mauradewey Mar 12, 2026
cd4af85
adding FAIR GSAT file to reproduce scaling issue
mauradewey Mar 12, 2026
fc591ca
Fix to ensure path exists for netcdf dump
maritsandstad Mar 13, 2026
254b265
Various fixes for scaling timeseries and slightly more varried input …
maritsandstad Mar 13, 2026
d8fdeab
Merge pull request #69 from benmsanderson/fastmip-scaling-error-explore
mauradewey Mar 16, 2026
86fecdd
FastMIP phase 2 notebook to train METEOR and generate requested output
mauradewey Mar 19, 2026
95e4345
cleanup phase 2 notebook
mauradewey Apr 1, 2026
b3ef982
splitting fastmip notebook into scripts. adding conda environment for…
mauradewey Apr 19, 2026
9a1157c
cleaning up plotting notebook
mauradewey Apr 20, 2026
5244d5f
stop tracking big notebook, fastmip output now created with scripts (…
mauradewey Apr 20, 2026
24174b7
updated pipeline with latest FAIR output
mauradewey Apr 22, 2026
7e6a470
updating fastmip pipeline
mauradewey May 27, 2026
0adbf10
script to update netCDF attributes for fastmip files
mauradewey May 27, 2026
983b428
results uploaded to ETH, code changes include updating file attribute…
mauradewey Jun 1, 2026
22f40e4
weird plot tests for FASTMIP and precip issue in gridded annual output
mauradewey Jun 9, 2026
0d25acf
Merge pull request #75 from benmsanderson/base
mauradewey Jun 10, 2026
75759b3
Preserve low-frequency global variability: default to pure VAR (use_e…
benmsanderson Jun 11, 2026
4f5df80
Fixing scaling
maritsandstad Jun 11, 2026
25f0686
Tests, fixes for more diverse data
maritsandstad Jun 12, 2026
b65d678
Merge pull request #78 from benmsanderson/noise-temporal-persistence
mauradewey Jun 12, 2026
ae868ef
Merge pull request #77 from benmsanderson/fix-extended-timescaling
mauradewey Jun 12, 2026
e16947b
interface field generation fix on base
maritsandstad Jun 13, 2026
402e20b
Linting, Changelog
maritsandstad Jun 13, 2026
53bcf4f
Merge branch 'fastmip' into generation-cleanout-v2
mauradewey Jun 15, 2026
706a94a
Fit precipitation quantile map per month-of-year, full window
benmsanderson Jun 16, 2026
cd93443
Merge pull request #84 from benmsanderson/fix-precip-seasonal-transform
maritsandstad Jun 17, 2026
2ad4f85
linting, notebook update
maritsandstad Jun 29, 2026
a11f269
Fix offending docstring
maritsandstad Jun 29, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ __pycache__
.env
.cache/
cache/
data/
fastmip_env/

#Tempfiles
*~
Expand Down
11 changes: 11 additions & 0 deletions CHANGELOG.rst
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,17 @@ The changes listed in this file are categorised as follows:
[Unreleased]
---------------------

### Fixed
- Fixes to generate annual and monthly gridded ensembles with unified noise and preserving more of the variance.

### Changed

- Now possibly to send variable length temperature scaling timeseries, fixed noise generator for wrong ordering of base data dimensions

### Fixed

- Variable length timeseries now works also when don't have "year" as time dimension.


[Version 1.6.0]
-----------------------------
Expand Down
44 changes: 44 additions & 0 deletions FastMIP_README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@

Steps to produce the FastMIP Phase 2 Tier 1 METEOR contributions.
April 2026 by Maura Dewey (maura.dewey@cicero.oslo.no)

1. Train METEOR for all the models and scenarios. This is done with the METEOR venv (for example, within a tmux session on nac)
a. cd to METEOR folder and activate venv with: "source venv/bin/activate"
b. cd to scripts folder and run: "python FastMIP_phase2_trainMETEOR.py > output.txt 2>&1" - The list of ESMs that are emulated, the scenarios that are run, and the choices for subsetting the FAIR ensemble are made in the top of this python script. (NOTE: If you need to change the training settings, ie. which scenario or ESM, it needs to be done in this script. Possible update/to-do would be to put that in a yml file.)
c. Disconnect from the tmux session and let it run. This will train METEOR and create all the raw METEOR output for the given list of ESMs and scenarios (takes ~10 hrs for Tier 1, if also training new METEOR instances)

2. Create FastMIP specific output files. This is done with a conda env fastmip_env
a. setup modules, deactivate METEOR venv, and activate conda environment with: ". FastMIP_env_setup.sh"
b. cd to scripts folder and create FastMIP specific output files with: "python FastMIP_phase2_makeoutput.py"
c. disconnect from the tmux session and let it run (takes a couple hours).
d. Once the output is done, update netCDF attributes with: "python FastMIP_phase2_atts.py"

3. Create plots
The notebook FastMIP_phase2_plots.ipynb will create all the plots from the Fastmip phase2 example notebook (https://github.com/sarasita/fastMIP/blob/main/requested_output/tour_through_requested_output.ipynb) with METEOR results.


Files on fastmip branch:

./FastMIP_README.md
./conda_env_fastmip.yml (conda environment file for processing output, needed to use for python-cdo for regridding)
./FastMIP_env_setup.sh (to clear modules and activate conda environment on nac server)

./scripts/FastMIP_phase2_trainMETEOR.py (trains METEOR for all Tier 1 models and creates 200 member ensembles - 20 FAIR ensemble members x 10 METEOR noise model realizations. The subselection of FAIR members used is saved in a pickle in ./data/FASTMIP_phase2/FAIR_data)
./scripts/FastMIP_phase2_makeoutput.py (makes requested FastMIP output files - subset 10 member ensemble and bulk statistics across full 200.)
./scripts/FastMIP_phase2_attrs.py (updates netCDF attributes to FastMIP protocol)

./notebooks/FastMIP_phase2_plots.py (makes all plots as in https://github.com/sarasita/fastMIP/blob/main/requested_output/tour_through_requested_output.ipynb)


Data files are not tracked with git, but the following structure is required and/or created with above scripts:

./data/FASTMIP_phase2/FAIR_data (contains full FAIR ensemble files (climate_assessment_forced.csv), grid file for regridding (g025.txt), and the pickles of the FAIR subset.)
./data/FASTMIP_phase2/scenario_data (contains the emission and concentration data files needed for the SCM)
./data/FASTMIP_phase2/METEOR_emulations/raw (where METEOR output is saved initially)
./data/FastMIP_phase2/METEOR_emulations/aggregated (where regridded and combined tmp files are saved)
./data/FASTMIP_phase2/METEOR_emulations/processed (where FastMIP output is saved)

Initial emulations where done for 4 scenarios (L, M, H, VL)
The full CMIP7 scenario list and short_name markers are:
scenarios_list = ['SSP1 - Very Low Emissions', 'SSP2 - Low Emissions', 'SSP2 - Medium-Low Emissions', 'SSP2 - Medium Emissions', 'SSP3 - High Emissions', 'SSP5 - Medium-Low Emissions_a', 'SSP2 - Low Overshoot_a']
scenarios_short = ['VL','L','ML','M','H','HL','LN']
13 changes: 13 additions & 0 deletions FastMIP_env_setup.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
#!/bin/bash

# load modules and conda environment to run FastMIP_phase2_makeoutput.py, making the FastMIP phase 2 output files for METEOR.
# The same conda environment can be used as a kernel for FastMIP_phase2_plots.ipynb.
# Note: this will deactivate the METEOR venv!
# module versions as of 2026-04
# the environment can be created from conda_env_fastmip.yml

module purge
module load Anaconda3/2023.09-0
source deactivate
conda deactivate
conda activate /div/no-backup-nac/users/maurad/METEOR/fastmip_env/
16 changes: 16 additions & 0 deletions conda_env_fastmip.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
name: /storage/no-backup-nac/users/maurad/METEOR/fastmip_env
channels:
- defaults
dependencies:
- xesmf
- ipykernel
- matplotlib
- pandas
- dask
- cftime
- xarray
- cartopy
- netcdf4
- regionmask
- python-cdo
prefix: /storage/no-backup-nac/users/maurad/METEOR/fastmip_env
Binary file not shown.
Loading
Loading