functionality to create the .cdsapirc file for CDSAPI on demand from …

…credentials provided in the project folder. This will be useful when we don#t want to save the credentials in the container permanently

Signed-off-by: georg198 <[email protected]>

.gitignore

@@ -163,6 +163,7 @@ nsight-systems*
 build/
 mlruns/
 checkpoints/
+examples/weather/dataset_download/credentials/
 
 # Hydra
 outputs/

CHANGELOG.md

@@ -18,6 +18,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Changed
 
+- Automatic .cdsapirc file creation for /examples/weather/dataset_download/
+
 ### Deprecated
 
 ### Removed

examples/weather/dataset_download/README.md

@@ -15,6 +15,9 @@ flexibly select different meteorological variables for their training dataset.
     surface temperature variable however if you would like a more complete dataset such
     as the one used to train [FourCastNet](https://arxiv.org/abs/2202.11214),
     please use `conf/config_34var.yaml`.
+4. `credentials/cdsapi.yaml` - can be used to enter the credentials for CDS API when
+    using a container environment. Should never be commited because it contains
+    sensitive information.
 
 ## How to Use
 

examples/weather/dataset_download/credentials/cdsapi.yaml

@@ -0,0 +1,18 @@
+# SPDX-FileCopyrightText: Copyright (c) 2023 - 2024 NVIDIA CORPORATION & AFFILIATES.
+# SPDX-FileCopyrightText: All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+url:
+key:

examples/weather/dataset_download/era5_mirror.py

@@ -28,6 +28,7 @@
 import logging
 import numpy as np
 import fsspec
+import yaml
 
 urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
 
@@ -59,6 +60,42 @@ def __init__(self, base_path: str, fs: fsspec.AbstractFileSystem = None):
         self.metadata_file = os.path.join(self.base_path, "metadata.json")
         self.metadata = self.get_metadata()
 
+        # If .cdsapirc does not exist in home directory create it from source file in ./credentials
+        self.create_cdsapirc()
+
+    def create_cdsapirc(self):
+        """Creates the .cdsapirc file if it is not present in the home directory.
+        If provided it is using the information from ./credentials/cdsapi.yaml to create the .cdsapirc file automatically"""
+
+        home = os.path.expanduser("~")
+        cds_credentials_path = os.path.join(home, ".cdsapirc")
+
+        if not os.path.exists(cds_credentials_path):
+            source_file = os.path.join(
+                os.path.dirname(os.path.abspath(__file__)), "credentials", "cdsapi.yaml"
+            )
+
+            logger = logging.getLogger()  # Suppress logging from cdsapi
+            logger.warning(
+                """No .cdsapirc file found in the home directory. 
+                Please create it directly or alternatively specify it in %s"""
+                % source_file
+            )
+
+            # read credentials from source file in ./credentials/
+            with open(source_file) as stream:
+                credentials_dict = yaml.safe_load(stream)
+            assert (
+                credentials_dict["url"] is not None
+            ), "Please provide a valid CDSAPI url in cdsapi.yaml"
+            assert (
+                credentials_dict["key"] is not None
+            ), "Please provide a valid CDSAPI key in cdsapi.yaml"
+
+            # create .cdsapi in $HOME directory
+            with open(cds_credentials_path, "w") as target_file:
+                yaml.safe_dump(credentials_dict, target_file)
+
     def get_metadata(self):
         """Get metadata"""
         if self.fs.exists(self.metadata_file):