feat: revamp config loading by tobiasdiez · Pull Request #156 · pik-piam/remind-mfa

tobiasdiez · 2026-07-16T12:27:09Z

Migrate from seperate per-model config files to more comprehensive config files that can be layered.

In particular:

Replaced YAML config files with a single TOML configuration that has shared base settings plus per-material overrides. (No strong opinion about using toml, but its the 'pythonic' config format and Python has bultin support for reading such files)
Added a config loader that deep-merges across multiple config layers, and then afterwards folds the model section over the base config.
Slightly reworked the CLI runner to use Typer in order to support repeated --config options for layered overrides and --model to specify what model to run.
Using typer it was then also easly to add an interactive prompt when no arguments are provided to make Bennet happy.

Fixes https://github.com/pik-piam/industry_issues/issues/11.

Possible follow-ups:

The paths in the visualization and exports section are material-specific but actually follow the same pattern. We could remove those seperate config options by either using placeholders (eg figures_path = "data/%model%/output/figures) or use a non-configurable convention about the folder structure.
Make it possible to run multiple material models (at the moment either all or a single one can be selected)

for more information, see https://pre-commit.ci

tobiasdiez · 2026-07-16T12:30:51Z

 ]

-requires-python = ">= 3.10"
+requires-python = ">= 3.12"


Python 3.12 is needed for built-in toml loader and using the type alias keyword. There are other workarounds, but the Python upgrade was the easiest. The change is aligned with https://scientific-python.org/specs/spec-0000/, but not sure what Python version convention you guys want to follow.

Fine for me!

tobiasdiez · 2026-07-16T12:33:05Z

This is a draft for now as I still want to double-check that the config is effectively the same and the model runs result in the same output.

Early feedback, especially about the new default.toml config is nonetheless appreciated.

leonieschweiger · 2026-07-20T10:01:23Z

Looks good to me, thanks Tobias!
We could probably declutter the default.toml a bit by agreeing on a visualization and export set for all materials that we would like to plot/export regularly in the team, then the material-specific sections could be shortened and this would be the standard config to run in order to check changes for all materials. In addition, everyone could have their own .toml with user-specific settings (e.g. for me, plotting plastics in more detail).

JakobBD

This is great stuff, so helpful!!!
Thanks, it's greatly appreciated.

I have commented the point on yaml (Leonie and Bennet, please comment!), and a few nit-picky points of personal preference, and a few suggestions for further improvements.

JakobBD · 2026-07-20T14:06:48Z

    CEMENT = "cement"


+def get_model_classes():


For lazy loading, we could have this function take the model name as an argument, and only import the needed model? Maybe in a match/case?

JakobBD · 2026-07-20T14:08:03Z

@@ -1,7 +1,7 @@
 # %%
+from remind_mfa.common.helpers import ModelNames


While we're at it, this file could be deleted, along with those for the other materials.

JakobBD · 2026-07-20T14:23:54Z

+    selected_models = list(ModelNames) if selection == "all" else [selection]
+
+    for model in selected_models:
+        model_config = load_config(config_names, model)


There's currently no option to specify a config path other than cwd/config, and no option to give paths to individual config files.

I find that behaviour fine when prompting for config files, but at least I would find it more intuituve if

uv run run_remind_mfa.py --config config/default.yaml would not return an error.

Personally, I would change the behaviour such that --config always expects the full (relative) path, while prompting looks in the config folder (could even be relative to the repo instead of cwd).

I see the downside of having to give the full path several times for several config files, but the only use case for that right now is running on the cluster for input data generation.

JakobBD · 2026-07-20T14:25:03Z

 import logging
-import yaml
-import sys
+from typing import Annotated, Literal


While we're at it, rename file to remind_mfa.py? wdyt?

JakobBD · 2026-07-20T14:25:59Z

-def init_model(cfg: dict) -> CommonModel:
-    """Choose MFA subclass and return an initialized instance."""
+@app.command()
+def main(


To make that more convenient, you could add a (boolean) --remind-input option, which sets all the others (config files, materials, later also scenarios).

JakobBD · 2026-07-20T14:27:21Z

+    configure_logger()
+    selected_models = list(ModelNames) if selection == "all" else [selection]


I would move these two lines to main(), and pass selected_models to run_remind_mfa, only calling it models there.

JakobBD · 2026-07-20T14:33:38Z

@@ -0,0 +1,116 @@
+[base.input]


As discussed, I would personally prefer switching back to yaml - my apologies for the miscommunication and the resulting back and forth, that was my fault!

For our deep hierarchy and nesting (I count up to 6 levels), I find the yaml indentation much easier to read than the toml dot notation. For me that outweighs toml files being somewhat more pythonic.

I don't want to decide this alone. @leonieschweiger and @bennet21 - What do you think?

I also find the yaml indentation more clear and intuitive. If the advantages of toml are not big (which I cannot judge myself), I would also be in favour of keeping yaml, but I think I could adjust to both :)

@JakobBD We discussed this a bit. There is no strong preference from our side to either option. Yaml has the advantage of being familiar, toml that it optionally allows to extract groups of keys to a table. We discussed that eg sankey under visualization might be better as a table/indented subgroup, while the single-value options like extrapolation.do_visualize feel cleaner in the toml dot notation than with forced indention in yaml.

toml does allow indentation, and that would be our proposal (see last commit) but going back to yaml is definitely also a fine option for us.

JakobBD · 2026-07-20T14:34:47Z

 ]

-requires-python = ">= 3.10"
+requires-python = ">= 3.12"


Fine for me!

JakobBD · 2026-07-20T14:48:33Z

We could probably declutter the default.toml a bit by agreeing on a visualization and export set for all materials that we would like to plot/export regularly in the team

I agree! @leonieschweiger could you make a suggestion?

In addition, everyone could have their own .toml with user-specific settings (e.g. for me, plotting plastics in more detail).

I like the idea! We could add the folder to .gitignore, and except default.yaml and cluster.yaml?

JakobBD · 2026-07-22T10:00:34Z

+        consumption.per_capita = true
+        trade.do_visualize = true
+        sankey.do_visualize = true
+        sankey.plotter_args.slice_dict.e = "C"


Suggested change

sankey.plotter_args.slice_dict.e = "C"

sankey.plotter_args.slice_dict = {"t": 2050, "e": "C"}

JakobBD · 2026-07-22T10:44:26Z

+        consumption.do_visualize = false
+        consumption.per_capita = false
+        sankey.do_visualize = false
+        sankey.plotter_args.slice_dict.t = 2050


Suggested change

sankey.plotter_args.slice_dict.t = 2050

sankey.plotter_args.slice_dict = {"t": 2050}

tobiasdiez and others added 2 commits July 16, 2026 14:15

feat: revamp config loading

35a7ac3

[pre-commit.ci] auto fixes from pre-commit.com hooks

6b21ab5

for more information, see https://pre-commit.ci

tobiasdiez commented Jul 16, 2026

View reviewed changes

tobiasdiez requested review from JakobBD, bennet21 and leonieschweiger July 16, 2026 12:33

JakobBD reviewed Jul 20, 2026

View reviewed changes

use indention for config tables

0f12408

JakobBD reviewed Jul 22, 2026

View reviewed changes

Comment thread config/default.toml

JakobBD reviewed Jul 22, 2026

View reviewed changes

		@@ -1,7 +1,7 @@
		# %%
		from remind_mfa.common.helpers import ModelNames

		configure_logger()
		selected_models = list(ModelNames) if selection == "all" else [selection]

	sankey.plotter_args.slice_dict.e = "C"
	sankey.plotter_args.slice_dict = {"t": 2050, "e": "C"}

	sankey.plotter_args.slice_dict.t = 2050
	sankey.plotter_args.slice_dict = {"t": 2050}

Uh oh!

Conversation

tobiasdiez commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tobiasdiez commented Jul 16, 2026

Uh oh!

leonieschweiger commented Jul 20, 2026

Uh oh!

JakobBD left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JakobBD Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JakobBD Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JakobBD commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JakobBD Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tobiasdiez commented Jul 16, 2026 •

edited

Loading

JakobBD Jul 20, 2026 •

edited

Loading

JakobBD Jul 20, 2026 •

edited

Loading

JakobBD commented Jul 20, 2026 •

edited

Loading

JakobBD Jul 22, 2026 •

edited

Loading