Stuff that needs to be finished before we can create a new pre-prepared dataset

[JackKelly]

Issues that probable need to be done before we can start training ML models "properly":

• Train on 2020. Test on 2021. No validation set? #390
• Only use timesteps where Sun's elevation is > 10 degrees above horizon #373
• Update NWPDataSource with new NWP variable names #371
• Don't convert NWPs to float16! Converting to float16 results in infinities in dswrf #370
• Convert NWP grib files to Zarr intermediate #344
• Update Satellite Zarr reader for new format #313
• GSP's remove threshold #205 (only exclude GSPs with literally zero installed PV)
• NWPDataSource RuntimeError: Example is wrong shape! #410
• Move all data normalisation to nowcasting_dataloader #231
• Manager._get_first_batches_to_create() should set first batch to 0 if no batches are on disk #469
• Run prepare_ml_data.py this weekend #471

And some issues that aren't absolutely essential but are pretty important:

• prepare_ml_data.py should save separate logs to OutputData.filepath/<SplitName>/<DataSource> #156
• GSP xr.dataset has 'data' --> 'power_mw' #447
• MAKE A FEW BATCHES!
• In Manager use multiprocessing.Pool not ProcessPoolExecutor #325
• Speed up prepare_ml_data.py #341
• MAKE SOME MORE BATCHES!
• Put three A6000s back in.
• Compute means and standard deviation for new NWP Zarr #407
• GSP normalised in nowcasting_dataset #454
• GSP centroid not in the center #463
• batch v14 GSP: time #483
• NWPs aren't being created with the rest of the batches #482
• Sometimes the channel labelled "si10" (10 meter wind speed) has data from the hcct channel in it nwp#6
• -1s in the satellite data #484
• Why does leonardo claim to only have 94.3 GB RAM installed? #486
• Update config: v14 #488
• Create more batches (possibly after doing enable nowcasting dataloader to append to the CSV files to create more batches #485)
• Missing NWP data #494

Nice to have:

• Full test set (including all GSPs for all datetimes) #491
• Implement a start_date and end_date setting in the config YAML #425
• Run validation script at the end of prepare_ml_data.py? #317
• Document the contents & shapes of the NetCDF files that nowcasting_dataset outputs? #227
• Investigate why there are sometimes multiple GSPs at one location #272

[peterdudfield]

I like this summary issues you do - nice to get a good overview of things

[JackKelly]

Thanks! (Although I was also wondering if this list of issues actually be done using a github "milestone"... but really not sure what's best!)

[peterdudfield]

Ive added - #454 to this list

[peterdudfield]

ive added #483

[peterdudfield]

I'd be tempted to say this is done?

This milestone of getting a 'prepared ML dataset' seems complete. There are a few nice to haves left over.

[JackKelly]

I agree! Great work everyone! Let's mark this as "closed" now. Woo!