Merge pull request #50 from alliander-opensource/feature/converters-doc

markdown for converters

docs/conf.py

@@ -24,14 +24,9 @@
     "sphinx.ext.intersphinx",
     "numpydoc",
     "hoverxref.extension",
-    "myst_parser",
+    "myst_nb",
 ]
 
-source_suffix = {
-    ".rst": "restructuredtext",
-    ".md": "markdown",
-}
-
 templates_path = ["_templates"]
 exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]
 
@@ -71,3 +66,8 @@
 # -- sphinx.ext.intersphinx config -------------------------------------------
 # For linking to power-grid-model's documentation.
 intersphinx_mapping = {"power-grid-model": ("https://power-grid-model.readthedocs.io/en/stable/", None)}
+
+
+# -- Temporary config -----------
+# Disable notebook executions in examples directory
+nb_execution_excludepatterns = ["examples/*.ipynb"]

docs/converters/converter.md

@@ -0,0 +1,60 @@
+<!--
+SPDX-FileCopyrightText: 2022 Contributors to the Power Grid Model IO project <[email protected]>
+
+SPDX-License-Identifier: MPL-2.0
+-->
+
+# Converters
+
+There are 4 types of converters present as of now. 
+Here, we shall discuss their basic structure and guidelines for building a custom converter.
+
+Use the examples notebooks to understand how to convert data from the respective formats. 
+
+- **PGM JSON Converter:** Refer to the [PGM JSON Example](../examples/pgm_json_example.ipynb)
+- **VisonExcelConverter** and **GaiaExcelConverter:** Refer to the [Vision and Gaia Example](../examples/vision_gaia_example.ipynb)
+- **Pandapower Converter:** Converts [pandapower network](https://pandapower.readthedocs.io/en/stable/elements.html), which is a dictionary of dataframes, to power-grid-model data.
+
+Refer to [converters](../power_grid_model_io.md#converters) in API documentation for more details
+
+## Structure
+
+`VisonExcelConverter` and `GaiaExcelConverter` are inherited from [tabular converters](tabular_converter.md) for excel exports of vision and gaia respectively.
+All 4 converters are derived from the base {py:class}`power_grid_model_io.converters.base_converter`. 
+The usable functions for loading, saving and converting the data are located in the base class. 
+The private functions (`_load_data`, `_parse_data` and `_serialize_data`) are overloaded based on the specific type of converter (ie. excel, json or pandapower). 
+It is recommended to create any custom converter in a similar way.
+
+## Instantiation
+
+A converter object can be instantiated in the following way. For eg, for a `PgmJsonConverter`,
+
+```python
+from power_grid_model_io.converters.pgm_json_converter import PgmJsonConverter
+
+converter = PgmJsonConverter(source_file=source, destination_file=destination)
+```
+
+The usable methods of converters for loading and saving the data are described below.
+
+## Loading data
+
+Use the methods load_input_data(), load_update_data(), load_sym_output_data() or load_asym_output_data() to load the relevant data to the converter.
+The Converter can be initialised with `source_file` containing path to source data. Or alternatively, the data can be provided as an argument to the load function.
+
+In addition to the power-grid-model input data, other miscellaneous information in the source file not used in calculations by power-grid-model gets stored under `extra_info`
+
+```python
+input_data, extra_info = converter.load_input_data(data=example_data)
+```
+
+## Saving Data
+
+It is possible to save the data in the format of the converter.
+The Converter can be instantiated with a path given to `destination_file`. 
+Alternatively, the destination path can be provided in the save function.
+You can also add additional information about each component in the form of `extra_info` generated by [Load data](converter.md#load-data) to be saved along with it.
+
+```python
+converter.save(example_data, extra_info=example_extra_info, destination=destination_path)
+```

docs/converters/tabular_converter.md

@@ -10,7 +10,7 @@ each with multiple `columns`, possibly with a specific `unit`.
 Others may have a categorical value which needs to be mapped (i.e. open: 0, closed: 1); in general we'll call these 
 `substitutions`.
 
-# Mapping file
+## Mapping file
 A mapping file is a yaml with three main sections `grid`, `units` and `substitutions`:
 ```yaml
 grid:
@@ -146,7 +146,7 @@ The definitions above can be interpreted as:
     the word "none" should be replaced with the boolean `false`
     and the word "own" should be replaced with the value boolean `true`.
 
-# AutoID
+## AutoID
 The `id` field is special in the sense that each object should have a unique numerical id in power grid model. 
 Therefore, each id definition is mapped to a numerical ID.
 Also each field name that ends with `node` is converted into the matching numerical ID.
@@ -161,9 +161,9 @@ c = auto_id("Alpha")   # c = 0 (because key "Alpha" already existed)
 item = auto_id[1]      # item = "Bravo"
  ```
 
-See also [AutoID](/src/power_grid_model_io/utils/auto_id.py).
+See also {py:class}`power_grid_model_io.utils.AutoID`
 
-## Vision and Gaia
+### Vision and Gaia
 For Vision and Gaia files, an extra trick is applied. Let's assume this mapping:
 
 ```yaml

docs/examples/pgm_json_example.ipynb

@@ -0,0 +1,115 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# PGM JSON Converter\n",
+    "\n",
+    "Converts to and from the power-grid-model JSON data format. \n",
+    "This json format makes it convenient to evaluate the grid data in perspective of input to power-grid-model. \n",
+    "More details about the JSON format are mentioned in `power_grid_model.utils` and example of it is in [Make Test Dataset](power-grid-model:examples/Make%20Test%20Dataset.ipynb) in power-grid-model repository.\n",
+    "\n",
+    "Basic imports and configuration:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import logging\n",
+    "from pathlib import Path\n",
+    "\n",
+    "import structlog\n",
+    "from power_grid_model import PowerGridModel\n",
+    "\n",
+    "from power_grid_model_io.converters.pgm_json_converter import PgmJsonConverter\n",
+    "\n",
+    "DATA_DIR = Path(\".\") / \"data\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Define source and destination paths"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "input_file = DATA_DIR / \"tiny-net\" / \"input.json\"\n",
+    "output_file = DATA_DIR / \"tiny-net\" / \"sym_output.json\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Instantiate the converter, optionally with source and destination file paths. Use `load_input_data()` to convert to power-grid-model data. Additional information is stored in `extra_info`. Run powerflow calculation with the `input_data`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "2022-11-01 16:08.58 [debug    ] Loading PGM input data\n"
+     ]
+    }
+   ],
+   "source": [
+    "converter = PgmJsonConverter(source_file=input_file, destination_file=output_file)\n",
+    "input_data, extra_info = converter.load_input_data()\n",
+    "pgm = PowerGridModel(input_data=input_data)\n",
+    "output_data = pgm.calculate_state_estimation()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The converted data or loaded data can be saved. The file will be saved in `output_file` path or the destination path can be provided here optionally."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "converter.save(data=output_data, extra_info=extra_info)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.13"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

docs/examples/pgm_json_example.ipynb.license

@@ -0,0 +1,3 @@
+SPDX-FileCopyrightText: 2022 Contributors to the Power Grid Model project <[email protected]>
+
+SPDX-License-Identifier: MPL-2.0