Adding Météo-France radar precipitation dataset

[gzemo]

Describe your changes

Pull request to start reviewing the addition of Météo-France Radar dataset (from 2020-01-01 to 2024-12-31, 5min, 1km grid-size).

The complete .zarr converted archive will be hosted on the ECMWF EWC S3 bucket.

Conversion pipeline from raw .npz radar data to the stardardised (plus validated via the mlcast-validator) .zarr format will be shared in a dedicated mlcast-community repository.

Issue Link

Original issue (#49).

Type of change

• 📖 Documentation (Addition or improvements to documentation):
  logs and .yaml configuration files updated by now.

Checklist before requesting a review

• Complete minor validator requests to build the definitive mlcast-compliant .zarr archive.
• Add conversion code in a mlcast-community repository.
• Upload dataset to the object-storage.

[github-actions]

View preview of built jupyterbooks on https://mlcast-community.github.io/mlcast-datasets/pr-preview/pr-51/
(preview is automatically rebuilt and uploaded on later commits)

[gzemo]

Fixing minor typos on the bucket path to zarr file.

[gzemo]

Interestingly, it seems like that the final documentation notebook check passed because of a manual .ipynb edit. Specifically, after correcting each cell id field from the raw jupyter notebook file replacing random alphanumerical idxs to ordered numerical values, this newly-passed requirement seems like satisfied. Maybe a VSCode-related issue (?).

[gzemo]

Conversion reference repository to perform the file conversion available at: https://github.com/mlcast-community/mlcast-dataset-FR-MF

[gzemo]

Sharing the results from the mlcast-validator's checks.

[LBerth]

Hi !
Congrats on all the work that you did to include Meteo-France's dataset into MLCast ! 😊 It's really nice to see that it can be useful.

I just have 2 questions :

• In the notebook, would it be possible to include the link to the hugging face dataset ? (maybe I read too fast and missed it)
• Last week, at the MLCast community webinar, when you showed a map of precipitation on France, was the map created with the script in this notebook ? Because I think that the projection of the map was not the same as the projection of the data, and the coasts and borders lines were not correctly positioned. You can compare with the map we provided on HuggingFace. In the notebook, I see that you have called the correct projection string with cartopy. If the plot was produced with the notebook, then I fear that cartopy is not able to draw the correct coasts and borders, because the projection we use is very specific and not common. We had this issue when we added the dataset on HuggingFace and created some basic plot scripts. The solution we found was to first project the data to a basic PlateCarrée projection, and then make the plot with cartopy.

Don't hesitate if you need more information.
Have a nice day,
Léa for MF's AI Lab

[leifdenby]

Good suggestion about including a link to hugging face @LBerth! Do you want to add that @gzemo?

I also just pushed a commit to your branch @gzemo to make sure the notebook is included in the table-of-contents, otherwise it is not rendered (maybe we need a test to ensure all notesbooks are listed in the TOC...)

I just had a look at the rendered notebook (https://mlcast-community.github.io/mlcast-datasets/pr-preview/pr-51/), by eye the spatial extent looks quite similar to the one on huggingface @LBerth - where do you see the borders/coastlines being incorrect? One could using the plotting routine on huggingface to really double check, what do you think @gzemo?

[LBerth]

I just checked the rendered notebook and indeed it seems okay to me. 👍

The issue was in last week presentation, where you can see that the radar field is different near Brittany for instance :

[gzemo]

Hi @LBerth & @leifdenby!

Many thanks for your support and suggestions.

Following, some updates on my side regarding some additional details / implementation.

1. I have not specifically mentioned that we had to convert the original PROJ_WKT as in plots.py (line 25) from v1 to v2 in order to allow the lat/lon bounding box information to be stored to have the zarr georeferencing to be ultimately mlcast-validator compliant. The final adapted version can be found in scripts/zarr_init.py (line 26) for comparison. We have made sure that the v2 version with the BBOX addition is a valid WKT string, albeit a double-check might still be needed.
2. The initial notebook's markdown cell is now correctly referring to i) the dataset download/conversion repository, ii) the original Hugging Face dataset (as you were correctly suggesting) iii) the full citation and iv) the redistribution license. Please let me know If I am still missing something.
3. (Minor edits) While rendering the final plots with cartopy I had to downgrade both matplotlib==3.10.1 and cartopy==0.24.1 (from 3.11.0 and 0.25.0, respectively) in order to correctly show the lat/lon ticks. From the default mlcast-dataset pyproject.toml file the latest libraries versions are automatically downloaded and prevent the "old" plotting command to be fully working (check below). I have not pushed the pyproject.toml with the corresponding plotting libraries version yet: let me eventually know if needed.

for ax in g.axes.flat:  # .axes is deprecated in v3.11.0
    ax.coastlines()
    ax.gridlines(draw_labels=["top", "left"]) # draw_labels is deprecated in v0.25.0

In the last week presentation we had used the plotting code from the Hugging face repository. One can clearly see, as compared with the currently rendered one in the newly-added notebook, that latest plots show that the north most radar coverage is clearly including a broader portion of the southern UK coastlines.

[leifdenby]

1. I have not specifically mentioned that we had to convert the original PROJ_WKT as in plots.py (line 25) from v1 to v2 in order to allow the lat/lon bounding box information to be stored to have the zarr georeferencing to be ultimately mlcast-validator compliant. The final adapted version can be found in scripts/zarr_init.py (line 26) for comparison. We have made sure that the v2 version with the BBOX addition is a valid WKT string, albeit a double-check might still be needed.

Ah, I see :) When you say "v2" is that v2 of the plotting script that meteo-france shared or is it something about the WKT string format? Or are you just calling this "v2" because you had to add the bbox info?

2. The initial notebook's markdown cell is now correctly referring to i) the dataset download/conversion repository, ii) the original Hugging Face dataset (as you were correctly suggesting) iii) the full citation and iv) the redistribution license. Please let me know If I am still missing something.

Looks good to me, what do you think @LBerth?

3. (Minor edits) While rendering the final plots with cartopy I had to downgrade both matplotlib==3.10.1 and cartopy==0.24.1 (from 3.11.0 and 0.25.0, respectively) in order to correctly show the lat/lon ticks. From the default mlcast-dataset pyproject.toml file the latest libraries versions are automatically downloaded and prevent the "old" plotting command to be fully working (check below). I have not pushed the pyproject.toml with the corresponding plotting libraries version yet: let me eventually know if needed.

Interesting! Do you think this is a bug in matplotlib / cartopy?

I will be on holiday for the next three weeks so I won't be able to look at this then. So maybe we will merge this back when I return, I hope that is ok :)

[gzemo]

Ah, I see :) When you say "v2" is that v2 of the plotting script that meteo-france shared or is it something about the WKT string format? Or are you just calling this "v2" because you had to add the bbox info?

I am actually referring to the WKT-CRS data standard version (i.e. the WKT string format). The CRS_WKT provided by Meteo-France as in the original Hugging face repository is in WKT v1. To the best of my knowledge, the only possible way we had to add the required spatial contextual information (BBOX) was to convert the original WKT string from v1 to v2 and then add the said field.

3. (Minor edits) While rendering the final plots with cartopy I had to downgrade both matplotlib==3.10.1 and cartopy==0.24.1 (from 3.11.0 and 0.25.0, respectively) in order to correctly show the lat/lon ticks. From the default mlcast-dataset pyproject.toml file the latest libraries versions are automatically downloaded and prevent the "old" plotting command to be fully working (check below). I have not pushed the pyproject.toml with the corresponding plotting libraries version yet: let me eventually know if needed.

Interesting! Do you think this is a bug in matplotlib / cartopy?

Not really: I simply downgraded both versions to be compliant with the previous notebooks' visualisation command due to some deprecated matplotlib syntax. I haven't properly investigated a possible troubleshoot to generate the exact same plot with the new matplotlib==3.11.0 ax.gridlines(...) method.

[LBerth]

Hi,
Thanks for the notebook's modifications, it's okay for me.
And thanks for the explanation on the WKT, I think we will also update our scripts on HuggingFace accordingly to use the v2.