Skip to content

Graph Pipeline with Multi-Backend, Spatial Indexing, and ML Integration #128

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Srishti-1806 wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Graph Pipeline with Multi-Backend, Spatial Indexing, and ML Integration #128

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
262b2b0
Notebook updated, ipynb improved
Srishti-1806 Mar 31, 2026
c25a12e
Merge upstream with resolved conflicts
Srishti-1806 Mar 31, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 78 additions & 1 deletion CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,84 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Added
### Added - Major New Features (v0.4.0 - Research-Grade Release)

#### 1. Backend Abstraction Layer
- **Multi-backend support**: NetworkX (debugging), PyTorch Geometric (training), DGL (high-performance)
- `GraphBackend` abstract base class with automatic format detection
- `NetworkXBackend`, `PyGBackend`, `DGLBackend` implementations
- `get_backend()` auto-detection for seamless format switching
- Enables scalable graph processing for weather models with 10⁵–10⁷ nodes

#### 2. Spatial Indexing Optimization
- KD-Tree based spatial indexing for O(log N) neighbor queries (vs O(N²) naive)
- `SpatialIndex`, `KDTreeIndex`, `BallTreeIndex` classes
- `create_spatial_index()` and `find_neighbors_vectorized()` functions
- **Performance**: 25-2500x speedup on large point sets (1K-10M points)
- Makes large weather grids (100K+ nodes) computationally feasible

#### 3. Temporal Graph Support
- `TemporalGraph` class for dynamic graphs with temporal edges
- `create_temporal_graph()` for autoregressive time-series modeling
- `add_temporal_edges_to_graph()` for manual temporal connection
- Enables recurrent neural network architectures and weather prediction
- **Feature**: Supports variable temporal window (history length)

#### 4. Configuration-Driven Pipeline
- `GraphConfig` dataclass for declarative graph definition
- YAML-based graph configuration for reproducibility
- `PipelineBuilder` for config-driven graph creation
- `create_graph_from_config()` for loading YAML/dict configs
- Example configs: `config_keisler.yaml`, `config_graphcast.yaml`

#### 5. Feature Engineering Layer
- `FeatureExtractor` class with specialized weather feature functions
- `add_wind_velocity()`: Magnitude from u/v components
- `add_wind_direction()`: Computing wind direction angles
- `add_pressure_gradient()`: Spatial pressure changes
- `add_temporal_encoding()`: Sinusoidal temporal positional encoding
- `add_spatial_encoding()`: Positional encoding based on coordinates
- `add_node_degree_features()`: Graph topology features
- `normalize_features()`: Min-Max and Z-score normalization

#### 6. ML Pipeline Integration
- `GraphDataset` and `PyGGraphDataset` for PyTorch compatibility
- `create_dataloader()`: Easy PyTorch DataLoader creation
- `create_pyg_dataloader()`: Specialized PyTorch Geometric DataLoaders
- `batch_graphs()`: Vectorized batching of multiple graphs
- `create_model_input()`: Format conversion for different ML frameworks
- `split_graph_for_training()`: Train/val/test node splitting

#### 7. Prebuilt Graph Architectures
- `load_prebuilt()`: Quick access to common architectures
- `list_prebuilt()`: Discover available architectures
- `register_archetype()`: Register custom architectures
- Implemented architectures:
- **Keisler (2021)**: Single-scale flat mesh (simple, interpretable)
- **GraphCast (Lam et al., 2023)**: Flat multiscale mesh (state-of-the-art)
- **MeshGraphNet (Pfaff et al., 2021)**: Hierarchical mesh (powerful)

#### 8. Enhanced Dependencies
- New optional dependencies for specific features:
- `[pytorch]`: PyTorch + PyTorch Geometric
- `[dgl]`: DGL framework support
- `[config]`: YAML configuration support (PyYAML)
- `[ml]`: Full ML pipeline (torch, torch-geometric, scikit-learn)
- `[visualisation]`: Plotly for advanced visualization
- `[all]`: All optional dependencies

#### 9. Examples and Documentation
- 6 comprehensive examples demonstrating all major features:
- `example_1_prebuilt.py`: Quick start with prebuilt graphs
- `example_2_config.py`: Configuration-driven creation
- `example_3_temporal.py`: Temporal graphs for time series
- `example_4_features.py`: Feature engineering workflow
- `example_5_spatial_indexing.py`: Performance optimization
- `example_6_ml_pipeline.py`: ML integration and DataLoaders
- Example YAML configurations for different architectures
- Expanded README with detailed use cases and quick starts

### Added (Previous Unreleased)

- Added a standalone graph consistency checking tool (`wmg.diagnostics.check_graph_consistency`) to ensure structural health, such as verifying all grid nodes successfully connect to the mesh (#42).
- Add Django-style graph filtering via `filter_graph`, for example to select
Expand Down
Loading