Big new design part 1 :)

[JackKelly]

Pull Request

Description

Imlement roughly the first half of the "Big New Design"! This is quite a big PR, sorry, because it's plugging together the new code.

Broadly implements an updated version of the design first sketched out in #213 (comment)

Also implements / fixes some other issues which were blocking this PR:

• load_solar_pv_data_from_gcs() should use fsspec and hence be able to load data from any compute environment #286
• Assert there's no overlap between train, test and validation datetimes at end of split() function #299
• Change DataSourceList into Manager; and maintain DataSources in a dict instead of a list? #298
• Allow user to configure the frequency of the t0 datetimes in the config yaml #277
• Pass command-line-arguments into prepare_ml_data.py #171

How Has This Been Tested?

The new prepare_ml_data.py runs succesfully.

But the unit ests are currently failing (deliberately! I haven't updated the tests yet!)

• No
• Yes

Checklist:

• My code follows OCF's coding style guidelines
• I have performed a self-review of my own code
• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works
• I have checked my code and corrected any misspellings

[JackKelly]

Hi @jacobbieker. As we discussed yesterday, I've sketched out the very rough design in main() in prepare_ml_data.py. The code is pretty broken at the moment so please don't pay any attention to the details yet! But, if you get a minute, please do take a look at the broad sequence of steps sketched out in main() in prepare_ml_data.py and let me know if you have any comments! Thanks!

[jacobbieker]

LGTM! I like the plan and the simplicity of it. Just one comment

[jacobbieker]

If the paths exist, but this doesn't, how would the script know what the spatial and temporal locations of each example is? Should this throw an error if the paths exist, but this doesn't? I guess, I think think this should go before the getting the max batch ID, and error out if its not overwrite, but batches do exist, and this file does not.

[JackKelly]

Very good point! As you suggested, I've moved "Check if the spatial_and_temporal_locations_of_each_example.csv files exist. If not, create these files." above checking for max_batch_id. Thanks! Good spot!

[JackKelly]

Hi @jacobbieker OK, I think I'll stop here in this PR; and continue in a subsequent PR tomorrow!

This PR implements a rough draft of (almost) all the steps listed in prepare_ml_data.main() up to and including creating spatial_and_temporal_locations_of_each_example.csv for each split.

prepare_ml_data.py runs. But the unittests still fail, and there are a bunch of linter errors, and I need to write a bunch of new unittests. But, if you fancy it, please do skim-read the code to make sure you're happy with the broad direction (but please don't worry about linter errors, missing docstrings etc... I'll get to those tomorrow, hopefully!)

Tomorrow, I'll start a new PR which builds off this one, and implements the second half of the "big new design": Reading in the spatial_and_temporal_locations_of_each_example.csv files and starting separate processes for each DataSource to prepare batches.

[jacobbieker]

Hi @jacobbieker OK, I think I'll stop here in this PR; and continue in a subsequent PR tomorrow!

This PR implements a rough draft of (almost) all the steps listed in prepare_ml_data.main() up to and including creating spatial_and_temporal_locations_of_each_example.csv for each split.

prepare_ml_data.py runs. But the unittests still fail, and there are a bunch of linter errors, and I need to write a bunch of new unittests. But, if you fancy it, please do skim-read the code to make sure you're happy with the broad direction (but please don't worry about linter errors, missing docstrings etc... I'll get to those tomorrow, hopefully!)

Tomorrow, I'll start a new PR which builds off this one, and implements the second half of the "big new design": Reading in the spatial_and_temporal_locations_of_each_example.csv files and starting separate processes for each DataSource to prepare batches.

Yeah, this looks great! I like how its set up and going!