Full test set (including all GSPs for all datetimes)

[peterdudfield]

In order to compare with ESO - do we need a test set, that is ALL GSPs for all datetimes?

Detailed Description

To compare against ESO we might need predictions from models for all gsps and all datetimes. This would be roughly
338 * (8 * 31 * 24 * 12) = 24 million predictions

Context

We want to compare our models to the ESO ones

Possible Implementation

• make batches for all timesstamps.
• just make 10,000 and comare ESO against them?

[peterdudfield]

@JackKelly @jacobbieker thoughts?

[JackKelly]

Ah, yes, very good point!

To do an exhaustive comparison against ESO's GSP-level PV forecasts, you're right that we should really run inference for every timestep of the test period (basically most of 2021). And, as you say, nowcasting_dataset doesn't currently do that: It currently produces batches of random examples (and doesn't guarantee full coverage over the test period).

But, during ML model training, we also want a smallish validation set, so we can look for overfitting during training.

It shouldn't be too hard to get Manager to produce a "full test set".

But let's set this as a lower priority that some of the other issues (like fixing NaNs) because, if worse comes to worst, we can still evaluate against ESO's forecasts, even with our current random test set

[JackKelly]

I've added this to the "nice to have" list of #393 but feel free to edit!

[jacobbieker]

Yeah, I agree with @JackKelly, once we have other stuff finished, then make the whole test set, so we know the data is looking good!

[JackKelly]

As Peter pointed out in openclimatefix/nowcasting_utils#72 we need a test set that includes all GSPs for all datetimes in order to compute national PV

[peterdudfield]

#515 might be good also only to do up to 2021-09-01 for the moment too

[peterdudfield]

Perhaps a different method would be to make sure the test set has all GSP's but doenst cover all datetimes.

Then we could do say 1000 datetimes. Total would be
1000 * 338 / 32 = ~ 10,000 batches

This seems a better way to start than the ~ 1 million batches for all datetimes