Skip to content
This repository was archived by the owner on Sep 11, 2023. It is now read-only.

Turn off temporal interpolation of numerical weather predictions (NWPs) #135

Closed
Tracked by #213
JackKelly opened this issue Sep 17, 2021 · 1 comment · Fixed by #338
Closed
Tracked by #213

Turn off temporal interpolation of numerical weather predictions (NWPs) #135

JackKelly opened this issue Sep 17, 2021 · 1 comment · Fixed by #338
Assignees
Labels
data New data source or feature; or modification of existing data source enhancement New feature or request

Comments

@JackKelly
Copy link
Member

JackKelly commented Sep 17, 2021

Detailed Description

The raw numerical weather predictions (NWPs) are hourly. But nowcasting_dataset interpolates the hourly data to produce 5-minutely NWPs in the pre-prepared batches.

Interpolating the NWPs massively blows up the filesize of the batches and - in theory (!) - this interpolation shouldn't be necessary for 'fully attentional' ML models like Perceiver, which don't need the inputs to all be aligned in time.

Context

"fully attentional" models view their inputs as a set. To tell the model about the position of each row of input data, we manually encode the position in time and space. As such, we should be able to feed in hourly NWPs and 5-minutely satellite images and PV data, and the model should learn how these things relate to each other in time and space.

Implementation

The implementation should be quite simple, because the code already handles inputs at different sample periods after the GSP pull request. We might need to make sure the NetCDFDataset correctly interprets the new batches.

@JackKelly
Copy link
Member Author

JackKelly commented Oct 29, 2021

Implemented in PR #307

Edit: And then re-enabled! Let's fix this in a subsequent PR!

Repository owner moved this from Todo to Done in Nowcasting Oct 29, 2021
@JackKelly JackKelly changed the title Allow user to turn off temporal interpolation of numerical weather predictions (NWPs) Turn off temporal interpolation of numerical weather predictions (NWPs) Oct 29, 2021
@JackKelly JackKelly reopened this Nov 4, 2021
Repository owner moved this from Done to In Progress in Nowcasting Nov 4, 2021
@peterdudfield peterdudfield self-assigned this Nov 5, 2021
Repository owner moved this from In Progress to Done in Nowcasting Nov 8, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
data New data source or feature; or modification of existing data source enhancement New feature or request
Projects
No open projects
Status: Done
Development

Successfully merging a pull request may close this issue.

2 participants