feat: Update Scenario Content, Structure, and Handling (and some other improvements)

[bennet21]

(Contains all changes from #155)

Purpose

This is a suggestion for a refactor of the scenario extrapolation implementation and represents a step towards more unified extrapolation handling in REMIND MFA.

Scenario Content

Scenario Inheritance

Scenarios now inherit from a BASE.csv that contains standard settings. Essentially all settings that were previously found in SSP2 are now found in there.

Scenario Structure

New Scenarios

I added some new scenarios (SSP3-5 and SSP1-2 CE) with a first estimate of their top-down parameter adjustments (Saturation level, lifetime).

Generalizing driver scenario S

With input data gen 1.6.0, floorspace became driver scenario dependent. This required a generalization of how this dimension is handled: gdp_pop_scen is now called driver_scen and every parameter that contains S will be sliced accoriding to the selected scenario (see common_model).

Generalized make_mfa (in common_model)

This is needed to make a bottom-up mfa in the cement_model. Everything stays backwards-compatible.

Scenario Handling

Example

The parameter emission_capture_rate in plastics had a target value and a target year in the scenario csv. However, the config yaml set it to constant extrapolation, so the scenario values were not used. This makes scenario and config ambiguous - it is unclear which information is used. With the proposed extrapolation approach, the model definition only selects which parameters are extrapolated, and some modelling decisions (type, blending, split handling). The scenario decisions are now unambiguously stored in the scenario csvs.
@leonieschweiger the target row (0.5 by 2050) was removed from the scenario csv in #155, so the rate is currently held constant. If the row is re-added, the new approach will apply it: the emission_capture_rate then increases to 0.5, which creates some negative flows - so lmk what you intended.

Proposed Working Principle

The core working principle is maintained.

Parameter Definition

In the model definition, you set up scenario parameters that are defined in the scenario.csvs. As before, there are
PlainDataPointDefinition which allows you to read in a single data point (currently used for driver scenario and saturation level)
ExtrapolationDefinition which allows you to read in whole arrays of data, which are converted into a flodym array and stored as a structured extrapolation instruction under scenario_parameters[name]. Let's walk through the settings

1. You provide a name
   If the name mirrors a name of a parameter that - at the time of extrapolation - already exists in your MFA parameters, the extrapolator knows you want to use it to extrapolate this parameter. If the name doesn't match anything but you instead want to create a new parameter and extrapolate it directly, you can specify
2. create_new
   This defaults to False. But it can be useful to initiate a new parameter here directly which you can use to anywhere in your MFA. This is currently used to create the parameters["stock_factor"] which is used to scale the stock after stock-extrapolation.
3. type
   Here, you provide how the extrapolation should use your scenario parameter. Possible settings are either factor or target. Factor means that the extrapolation method takes your parameter and multiplies it by a value given by the scenario parameter to create an extrapolated parameter. Target means that the parameter blends from the last historic value to the scenario parameter value.
4. blending_function
   As before, you can specify how the transition happens. The extrapolation is now always based on a blending from the old parameter to the values of (or multiplied by) the scenario parameter. The blending_function determines the shape of it, defaulting to a linear transition.
   5./6. split_dimension_letter and split_receiver_item may be niche applications
   With the first, you can provide a dimension letter along which a split is supposed to sum up to one. Then, the extrapolation renormalizes. E.g., if you increase your share of reinforced concrete buildings, it will proportionally decrease your share of other building structures, i.e., wood, steel and masonry.
   If you want the whole shift to reinforced concrete to happen through another coordinate, you can provide a receiver item. For instance, you could specify M (masonry) here, and all the growth in reinforced concrete buildings will be accompanied by a corresponding drop in masonry buildings.

A few more details.

• If you just want a constant extrapolation (hold last historic value constant), you can simply omit all entries except for name, see e.g. ExtrapolationDefinition(name="material_shares_use_inflow").
• The scenario parameter name has to match one parameter at the time of extrapolation (if create_new = False). So you can even extrapolate parameters that are not part of the read-in.

Scenario csv

The scenario csv stays largely the same (apart from the structural changes in #155). You just provide parameter, value and extra:year, as well as indices if desired and REMIND-MFA can connect it to the settings in the definition through the parameter name. Inheritance works as before.
For constant extrapolation, you don't even have to provide information in the scenario.csv (a value without extra:year has no effect and triggers a warning; an extra:year without a type in the definition raises an error).
But there is one larger improvement: instead of typing a number in the value column you can now mix-and-match with a parameter name in the form of a string. At extrapolation, data from that parameter will be used as extrapolation target or factor. You could e.g. prepare a world-average structure split parameter in mrmfa, read it in as a normal parameter and then provide its name in the scenario csv such that the structure split is converged towards the world-average split. This can be combined with any settings in the ExtrapolationDefinition. The world-average structure split parameter can even itself be a parameter that is extrapolated - in that case it is automatically extrapolated before it is applied for the extrapolation of the pure structure split.

Enabling Changes

The biggest changes are found in parameter_extrapolation. Here, the convoluted code has been combined into one class. This is why you don't have to provide any extrapolation class anymore. The previous base class ParameterExtrapolation is now the only necessary class. As before, it handles all possible dimension situations for incoming parameters: parameters that are constant in time, parameters that are only set historically, and parameters that already have a future trajectory. In its extrapolate method, the parameter is projected based on scenario settings: it either applies an absolute target or a relative factor. The _renormalize_split is applied when a split_dimension_letter is given. The _description provides some information for the assumption doc.

The ParameterExtrapolationManager works similarly to before, but now only passes ExtrapolationScenarioParameter instances on to extrapolation and orders them such that parameters used in other parameter extrapolations are extrapolated first.

Two new classes support this flow: ExtrapolationDefinition (in common_definition.py) is the structural class that lives in the model definition. ExtrapolationScenarioParameter (in scenarios.py) is the container created by ScenarioReader for each such definition - it carries the definition itself alongside the value array (object dtype, supporting string parameter-name endpoints alongside numbers), the is_set mask (distinguishing an explicit zero from an untouched entry), and the extras dict (e.g. year).

After extrapolation has been run, all extrapolation scenario parameter names are certain to be "real" parameters now - so you can use them via parameters[name].

Miscellaneous

• After extrapolation, parameters are rechecked with self.check_parameters() in the common model.
• data_blending has been refactored slightly for better external accessibility.
• The stock factor is now applied before the transition smoothing.

Selected Bottom-up improvements

Buttom-up parameters like building structure splits are now acting on the inflow to the stock, instead of the stock itself. This makes much more sense as it is what societies could change, and therefore what should be reflected in scenarios. To accomplish this, while all the splits are refering to the stock, we did the following. For the historical period, we assume a constant split of the inflow, which is equal to the split of the stock. As long as the lifetimes are not dependent on the dimensions of the split, this ensures that the stock also follows this split, which is what we have data on. Going into the future, we can now gradually change these splits according to scenario assumptions. All of this requires a bit of back-and-forth calculations, which are now found in cement_mfa_system_bottom_up.py.

[JakobBD]

[partly read, partly skimmed through source code. I haven't followed every detail of the parameter extrapolation module]

Looks great!
Haven't reviewed the bottom-up PR yet, but this one looks good to merge to me.

[leonieschweiger]

Great work Bennet, thanks a lot! I just have one suggestion with regards to where to set the "type" of the extrapolation, let's discuss :)

[tobiasdiez]

From a code-perspective, looks good to me. Two small remarks in the code below. Moreover, I would suggest to add a basic test for the paramater extrapolation/blending. But none of that is super important, so feel free to ignore these suggestions and just merge.