Skip to content

Extend set_meta_from_data() to apply on different column - #986

Open
danielhuppmann wants to merge 7 commits into
IAMconsortium:mainfrom
danielhuppmann:feature/set-meta-from-data-2
Open

Extend set_meta_from_data() to apply on different column#986
danielhuppmann wants to merge 7 commits into
IAMconsortium:mainfrom
danielhuppmann:feature/set-meta-from-data-2

Conversation

@danielhuppmann

@danielhuppmann danielhuppmann commented Jul 8, 2026

Copy link
Copy Markdown
Member

Please confirm that this PR has done the following:

  • Tests Added
  • Documentation Added
  • Name of contributors Added to AUTHORS.rst
  • Description in RELEASE_NOTES.md Added

Description of PR

This PR (finally) implements a feature that a user can check where some function applied to a timeseries in data holds and use the corresponding value in a different column for the meta value.

I've done this (manually) multiple times for getting the year where peak-warming (i.e., maximum of the temperature timeseries) is reached.

A few items to discuss:

  • I chose "on" as argument name for the column on which to apply the "method", following the logic of groupby(by)... But I'm open to suggestions for alternative argument names
  • The resulting meta-value is not necessarily unique, like when the temperature maximum is reached in two years. I (arbitrarily) use the first item and write a warning to the log.
  • The implementation of apply_method() is very "hands-on", there may be more elegant approaches with lumpy...?
  • For whatever reason, df.groupby()[column].apply("min") yields a different result from df.groupby()[column].apply(np.min) (because the latter evaluates element-wise).

Example

To set the year where peak-temperature is reached as meta-indicator "Year of Peak Warming":

df.set_meta_from_data(
    name="Year of Peak Warming",
    variable="Climate Assessment|Surface Temperature (GSAT)|Median",
    method="max",
    column="year",
    on="value",
)

@danielhuppmann
danielhuppmann requested a review from phackstock July 8, 2026 08:21
@danielhuppmann danielhuppmann self-assigned this Jul 8, 2026
@codecov

codecov Bot commented Jul 8, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.00000% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 94.7%. Comparing base (144a9eb) to head (633b63c).

Files with missing lines Patch % Lines
pyam/core.py 89.4% 2 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##            main    #986     +/-   ##
=======================================
- Coverage   94.7%   94.7%   -0.1%     
=======================================
  Files         70      70             
  Lines       6717    6732     +15     
=======================================
+ Hits        6365    6379     +14     
- Misses       352     353      +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@phackstock phackstock left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. Just a few minor comments and questions below.
On the topic of using on for the name of the column that is used to populate the values in the meta table, I think using on is good. I always like taking design patterns, or variable names in this case, from other, established libraries.

Comment thread pyam/core.py
Comment thread pyam/core.py Outdated
Comment thread pyam/core.py Outdated
Comment thread pyam/core.py Outdated
Comment thread pyam/core.py
value = x[x[on] == x[on].apply(method)][column].unique()

if len(value) > 1:
logger.warning(f"Non-unique result from {method} on column {on}.")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this just be a warning? I'd imagine that producing a non-unique value for a meta value can only be ill-conceived. We might want to raise an actual error here.

@danielhuppmann danielhuppmann Jul 17, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In my original use case, it could be that the temperature stabilises at (for example) 1.7°,, in particular if we round the output from the climate assessment. So there could be multiple years where peak temperature is reached, without this being necessarily incorrect.

The truly correct solution would be to also provide an on_method argument that makes this unique?

Comment thread tests/test_feature_set_meta.py Outdated
danielhuppmann and others added 2 commits July 17, 2026 18:16
Co-authored-by: Philip Hackstock <20710924+phackstock@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants