refactor variable scaling, pressure level scalings only applied in specific circumstances #52
Conversation
|
Please consider using the knowledge about variables that come from the dataset metadata. See https://github.com/ecmwf/anemoi-transform/blob/7cbf5f3d4baa37453022a5a97e17cc71a5b8ceeb/src/anemoi/transform/variables/__init__.py#L47 |
We have given this some thought, and after wanting to use the information from the dataset in the beginning, I have opted for allowing the definition of our own groups here to use different scaling for self-defined groups. |
| data_indices, | ||
| ).get_variable_scaling() | ||
|
|
||
| # Instantiate the pressure level scaling class with the training configuration |
There was a problem hiding this comment.
I don't know if this is possible but I wonder if here we could instantiate from a list of scalars rather than specific ones? I think this is how it is done for the validation metrics
There was a problem hiding this comment.
Yes, this would be useful, as we want to allow more scalar methods for model levels/tendency/etc.
I will have a look.
|
|
||
| # Instantiate the pressure level scaling class with the training configuration | ||
| pressurelevelscaler = instantiate( | ||
| config.training.pressure_level_scaler, |
There was a problem hiding this comment.
I think this config location is wrong? It should be training.variable_loss_scaling.pressure_level_scalar
There was a problem hiding this comment.
Indeed, in the config file, the pressure_level_scalar should be defined directly in training as before.
…umstances' of https://github.com/ecmwf/anemoi-core into 7-pressure-level-scalings-only-applied-in-specific-circumstances
sahahner
changed the title
|
|
| @abstractmethod | ||
| def get_variable_scaling(self) -> np.ndarray: ... | ||
|
|
||
| def get_variable_group(self, variable_name: str) -> tuple[str, str, int]: |
There was a problem hiding this comment.
I think maybe we should put this function in an utils. I only say this because I want to use it in another application outside of the loss scalings
|
Hi, I would like to know what you think about making all scalers explicit in the config file. Something similar to the |
Solve the problem explained in issue #7 by refactoring the variable scalings into a general variable scaling and a pressure level scaling.
@mc4117 , @pinnstorm and me came up with a new structure. This PR implements this.
This is first draft. Feedback very welcome!