Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat/migrate-graph-creation-to-torch #56

Draft
wants to merge 69 commits into
base: develop
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
69 commits
Select commit Hold shift + click to select a range
7826270
feat: move edges funcs to torch
JPXKQX Sep 18, 2024
0fd8611
feat: time edge builder
JPXKQX Sep 18, 2024
612d7ba
feat: moving things to torch
JPXKQX Sep 19, 2024
70d6bbf
fix: typo
JPXKQX Sep 19, 2024
b23bdb0
fix: dims
JPXKQX Oct 1, 2024
ad0abef
feat: torch-cluster dependency
JPXKQX Oct 1, 2024
15e8e09
Merge branch 'develop' into feature/torch-support
JPXKQX Oct 3, 2024
cf69280
fix: update CHANGELOG.md
JPXKQX Oct 3, 2024
bb39b61
fix: create BaseEdgeAttributeBuilder
JPXKQX Oct 7, 2024
0a99888
fix: expand dims only if ndim = 1
JPXKQX Oct 8, 2024
dedfe71
feat: move edge direction computation to torch.Tensor
JPXKQX Oct 8, 2024
bfe7be2
feat: add azimuth
JPXKQX Oct 8, 2024
35103c1
fix: update changelog
JPXKQX Oct 8, 2024
09ceba8
feat: support GPU
JPXKQX Oct 8, 2024
c32a39c
feat: log execution times in debug mode
JPXKQX Oct 8, 2024
e42bca3
feaet: unit-range added
JPXKQX Oct 18, 2024
8382588
fix: trigger readthedocs CI
JPXKQX Nov 25, 2024
e74747e
Expand index page with general overview
bluefoxr Nov 25, 2024
25b3ad8
Remove section links list from index page
bluefoxr Nov 25, 2024
2925028
feat: expand plot_nodes to multi-dimensional attributes
JPXKQX Nov 26, 2024
e4d7115
fix: update changelog
JPXKQX Nov 26, 2024
4a1ce3f
further reorg of toctree
bluefoxr Nov 26, 2024
5a92030
change overview to background, edits
bluefoxr Nov 26, 2024
496f010
merge CLI pages into one
bluefoxr Nov 26, 2024
5a0a15e
Merge branch 'develop' into feature/expand-plot-nodes
JPXKQX Nov 26, 2024
e156afe
fix: changelog
JPXKQX Nov 26, 2024
dc798ed
Merge branch 'develop' into docs/refactor
JPXKQX Nov 26, 2024
accd8aa
fix: changelog
JPXKQX Nov 26, 2024
8dcc389
expand CLI intro description
bluefoxr Nov 27, 2024
4fac665
move further details from graphs intro
bluefoxr Nov 27, 2024
9444533
further rearrangement of TOC
bluefoxr Nov 27, 2024
cd27027
heading
bluefoxr Nov 27, 2024
0565696
small edit
bluefoxr Nov 27, 2024
da96065
expanded intro to graph building
bluefoxr Nov 27, 2024
637d93c
tweak
bluefoxr Nov 27, 2024
c6bc645
tweak to coords
bluefoxr Nov 27, 2024
392c802
fix equations, formatting
bluefoxr Nov 27, 2024
f2ead1c
small edits to knn
bluefoxr Nov 27, 2024
2015ab5
add internal ref tag
bluefoxr Nov 27, 2024
be63607
add internal ref tag
bluefoxr Nov 27, 2024
f29ce15
another internal ref tag
bluefoxr Nov 27, 2024
fd0ea17
change heading, tweaks
bluefoxr Nov 27, 2024
eb548e7
numbers to text
bluefoxr Nov 27, 2024
7732b92
removed dots as may be confusing
bluefoxr Nov 27, 2024
1123781
Merge branch 'docs/refactor' of https://github.com/ecmwf/anemoi-graph…
bluefoxr Nov 28, 2024
86cc009
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 28, 2024
62e0dde
Merge branch 'develop' into docs/refactor
JPXKQX Dec 3, 2024
c197442
Merge branch 'develop' into feature/torch-support
JPXKQX Dec 4, 2024
1f6fad6
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 4, 2024
0d93a60
fix: code style
JPXKQX Dec 4, 2024
55a76da
docs: fix cli
JPXKQX Dec 4, 2024
ca9cb68
docs: move cli.rst
JPXKQX Dec 4, 2024
01ef0c3
docs: update cutout
JPXKQX Dec 4, 2024
b6f0ee4
docs: add post-processor
JPXKQX Dec 4, 2024
d753ab7
docs: more changes
JPXKQX Dec 4, 2024
77e42f9
Merge branch 'develop' into feature/torch-support
JPXKQX Dec 9, 2024
0873014
fix: merge torch-cluster dep
JPXKQX Dec 10, 2024
3d621f1
fix: torch-cluster optional dep
JPXKQX Dec 11, 2024
6a3cb82
Merge branch 'develop' into docs/refactor
JPXKQX Dec 11, 2024
d6bc06b
Merge branch 'develop' into feature/torch-support
JPXKQX Dec 11, 2024
7f6e425
fix: update tests
JPXKQX Dec 11, 2024
9f704b5
Merge branch 'feature/torch-support' of github.com:ecmwf/anemoi-graph…
JPXKQX Dec 11, 2024
0860ce1
sweep of typos
bluefoxr Dec 18, 2024
7e17c7e
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 18, 2024
fd7a961
Merge commit 'e156afe109d4da838cf8990fa57e837777df1f5c' into feature/…
JPXKQX Dec 20, 2024
35cd5d8
Merge commit '7e17c7e737f17a8c770d87636892f00219dbec34' into docs/ref…
JPXKQX Dec 20, 2024
bace801
Merge commit '9f704b51a959d9fa4b7882dcdb83a09ca42b782a' into feature/…
JPXKQX Dec 20, 2024
1282cd1
Merge branch 'develop' into feature/torch-support
JPXKQX Dec 31, 2024
5dfcd19
feat: split edge builders into different files
JPXKQX Dec 31, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion graphs/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,15 @@ Keep it human-readable, your future self will thank you!

### Changed

### Changed

- docs: Documentation structure (#84)
- fix: faster edge builder for tri icosahedron. (#92)

### Added

- feat: Support for multi-dimensional node attributes in plots (#86)

## [0.4.1 - ICON graphs, multiple edge builders and post processors](https://github.com/ecmwf/anemoi-graphs/compare/0.4.0...0.4.1) - 2024-11-26

### Added
Expand All @@ -33,7 +40,7 @@ Keep it human-readable, your future self will thank you!
- feat: Define node sets and edges based on an ICON icosahedral mesh (#53)
- feat: Support for multiple edge builders between two sets of nodes (#70)

# Changed
### Changed

- fix: bug when computing area weights with scipy.Voronoi. (#79)

Expand All @@ -46,6 +53,7 @@ Keep it human-readable, your future self will thank you!
- feat: New class, KNNAreaMaskBuilder, to specify Area of Interest (AOI) based on a set of nodes. (#30)
- feat: New node builder classes, LimitedAreaXXXXXNodes, to create nodes within an Area of Interest (AOI). (#30)
- feat: Expanded MultiScaleEdges to support multi-scale connections in limited area graphs. (#30)
- feat: New edge attribute, Azimuth. (#54)
- feat: New method update_graph(graph) in the GraphCreator class. (#60)
- feat: New class StretchedTriNodes to create a stretched mesh. (#51)
- feat: Expanded MultiScaleEdges to support multi-scale connections in stretched graphs. (#51)
Expand All @@ -59,6 +67,8 @@ Keep it human-readable, your future self will thank you!
### Changed

- ci: small fixes and updates pre-commit, downsteam-ci (#49)
- feat: Implementing KNNEdges and CutOffEdges with torch-cluster (#54)
- feat: Migrate edge attributes to inherit from MessagePassing (#54)
- Update CODEOWNERS (#61)
- ci: extened python versions to include 3.11 and 3.12 (#66)
- Update copyright notice (#67)
Expand Down
15 changes: 0 additions & 15 deletions graphs/docs/cli/create.rst

This file was deleted.

15 changes: 0 additions & 15 deletions graphs/docs/cli/describe.rst

This file was deleted.

16 changes: 0 additions & 16 deletions graphs/docs/cli/inspect.rst

This file was deleted.

73 changes: 61 additions & 12 deletions graphs/docs/cli/introduction.rst
Original file line number Diff line number Diff line change
@@ -1,29 +1,78 @@
.. _cli-introduction:

=============
Introduction
=============
=================
Command line tool
=================

When you install the `anemoi-graphs` package, this will also install command line tool
called ``anemoi-graphs`` which can be used to design and inspect weather graphs.
When you install the `anemoi-graphs` package, a command line tool will also be installed
called ``anemoi-graphs``, which can be used to build graphs based on YAML recipe files,
and inspect existing graphs.

The tool can provide help with the ``--help`` options:

.. code-block:: bash

% anemoi-graphs --help

The commands are:
To **create** a graph, use the ``create`` command:

.. toctree::
:maxdepth: 1
.. code:: console

create
describe
inspect
$ anemoi-graphs create recipe.yaml graph.pt

The ``.yaml`` recipe file consists of high-level specifications for generating the graphs at each
layer. An example of a simple recipe file is given in the :ref:`the following section <usage-getting-started>`.

The ``create`` command will read the specifications in the ``recipe.yaml`` recipe file, and write to a PyTorch
``.pt`` file.

To **describe** an existing graph stored as a ``.pt`` file, use the ``describe`` command:

.. code:: console

$ anemoi-graphs describe graph.pt

This will generate a text summary of the graph, including the number of nodes and edges
at each layer, the geographic boundaries, and statistics about the edge lengths:

.. literalinclude:: ../usage/yaml/global_wo-proc.txt
:language: console

A set of interactive and static visualisations are generated to allow visual inspection
of the graph design.

Finally, the ``inspect`` command will generate a set of interactive and static visualisations
for visual inspection of the graph design:

.. code:: console

$ anemoi-graphs inspect graph.pt output_folder/

===================
Command line usage
===================

Create Command
--------------

.. argparse::
:module: anemoi.graphs.__main__
:func: create_parser
:prog: anemoi-graphs
:path: create

Describe Command
----------------
.. argparse::
:module: anemoi.graphs.__main__
:func: create_parser
:prog: anemoi-graphs
:path: describe

Inspect Command
---------------
.. argparse::
:module: anemoi.graphs.__main__
:func: create_parser
:prog: anemoi-graphs
:nosubcommands:
:path: inspect
4 changes: 2 additions & 2 deletions graphs/docs/graphs/edge_attributes.rst
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@
Edges - Attributes
####################

There are 2 main edge attributes implemented in the `anemoi-graphs`
package:
There are two main edge attributes implemented in the
:ref:`anemoi-graphs <anemoi-graphs:index-page>` package:

*************
Edge length
Expand Down
20 changes: 11 additions & 9 deletions graphs/docs/graphs/edges/cutoff.rst
Original file line number Diff line number Diff line change
@@ -1,9 +1,11 @@
.. _cutoff_radius:

################
Cut-off radius
################

The cut-off method is a method for establishing connections between two
sets of nodes. Given two sets of nodes, (`source`, `target`), the
The *cut-off method* is a method for establishing connections between
two sets of nodes. Given two sets of nodes, (`source`, `target`), the
cut-off method connects all source nodes, :math:`V_{source}`, in a
neighbourhood of the target nodes, :math:`V_{target}`.

Expand All @@ -16,20 +18,20 @@ as,

.. math::

cutoff\_radius = cuttoff\_factor \times nodes\_reference\_dist
\text{cutoff_radius} = \text{cutoff_factor} \times \text{nodes_reference_dist}

where :math:`nodes\_reference\_dist` is the maximum distance between a
target node and its nearest source node.
where :math:`\text{nodes_reference_dist}` is the maximum distance
between a target node and its nearest source node.

.. math::

nodes\_reference\_dist = \max_{x \in V_{target}} \left\{ \min_{y \in V_{source}, y \neq x} \left\{ d(x, y) \right\} \right\}
\text{nodes_reference_dist} = \max_{x \in V_{target}} \left\{ \min_{y \in V_{source}, y \neq x} \left\{ d(x, y) \right\} \right\}

where :math:`d(x, y)` is the `Haversine distance
<https://en.wikipedia.org/wiki/Haversine_formula>`_ between nodes
:math:`x` and :math:`y`. The ``cutoff_factor`` is a parameter that can
be adjusted to increase or decrease the size of the neighbourhood, and
consequently the number of connections in the graph.
:math:`x` and :math:`y`. The :math:`\text{cutoff_factor}` is a parameter
that can be adjusted to increase or decrease the size of the
neighbourhood, and consequently the number of connections in the graph.

To use this method to create your connections, you can use the following
YAML configuration:
Expand Down
14 changes: 8 additions & 6 deletions graphs/docs/graphs/edges/knn.rst
Original file line number Diff line number Diff line change
@@ -1,11 +1,13 @@
.. _knn:

######################
K-Nearest Neighbours
######################

The knn method is a method for establishing connections between two sets
of nodes. Given two sets of nodes, (`source`, `target`), the knn method
connects all target nodes, to their ``num_nearest_neighbours`` nearest
source nodes.
The k-nearest neighbours (KNN) method is a method for establishing
connections between two sets of nodes. Given two sets of nodes,
(`source`, `target`), the KNN method connects all target nodes to their
``num_nearest_neighbours`` nearest source nodes.

To use this method to build your connections, you can use the following
YAML configuration:
Expand All @@ -21,5 +23,5 @@ YAML configuration:

.. note::

The KNNEdges method is recommended for the decoder edges, to connect
all target nodes with the surrounding source nodes.
The ``KNNEdges`` method is recommended for the decoder edges, to
connect all target nodes with the surrounding source nodes.
2 changes: 2 additions & 0 deletions graphs/docs/graphs/edges/multi_scale.rst
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
.. _multi_scale:

################################################
Multi-scale connections at refined icosahedron
################################################
Expand Down
Loading
Loading