The Problem
Currently, mllam-verification provides an excellent foundation for continuous, point-wise verification (rmse, mae, difference). However, for high-resolution Limited Area Models (LAMs) like those developed in neural-lam, point-wise metrics often suffer from the well-known "double penalty" effect. If the model perfectly predicts a storm's intensity and structure but displaces it slightly, it is penalized twice (once for missing the actual location, and once for predicting it where it shouldn't be).
Furthermore, as NWP shifts towards predicting extreme events, we currently lack threshold-based (categorical) verification metrics in the repository.
Proposed Solution
To make mllam-verification a complete diagnostic suite for high-resolution models, I propose adding a new suite of Spatial and Categorical verification metrics, leveraging xarray and the existing compute_pipeline_statistic infrastructure.
1. Spatial Verification: Fraction Skill Score (FSS)
Following Roberts and Lean (2008), FSS is the industry standard for evaluating high-resolution spatial forecasts. It compares the fractional coverage of events (e.g., precipitation > threshold) in spatial windows of increasing sizes.
- Metric: Implement
fss which computes the score for a given threshold and neighborhood size (radius).
- Plotting: Implement
plot_fss_scale() in plot.py to visualize FSS as a function of neighborhood size, allowing researchers to determine the spatial scale at which a model becomes "useful" (typically FSS > 0.5).
2. Categorical / Threshold Verification
Evaluate the model's ability to predict whether a variable exceeds a specific threshold (e.g., Wind Speed > 20m/s).
- Contingency Table: Implement a fast
xarray-based generator for Hits, Misses, False Alarms, and Correct Negatives.
- Metrics:
- Equitable Threat Score (ETS): For deterministic categorical forecasts.
- Brier Score (BS): For probabilistic forecasts (e.g., predicting the probability of an event).
Proposed Architecture & Integration
I plan to integrate these without breaking the current API:
- New Modules / Extensions: Add functions to
mllam_verification/operations/statistics.py or create dedicated spatial.py / categorical.py modules.
- Performance: Leverage
xarray.DataArray.rolling (or scipy.signal.convolve2d wrapped via xr.apply_ufunc) to compute fractional coverages efficiently across grid dimensions (x, y).
- Plotting: Extend
plot.py with plot_fss_scale (Line plot: Neighborhood Size on X-axis, FSS on Y-axis).
Core Reason
As neural-lam begins to integrate probabilistic models like Graph-EFM and CRPS-LAM, the ability to evaluate spatial structure and threshold exceedance becomes just as critical as continuous error. This contribution will provide the necessary tools for researchers to rigorously evaluate these next-generation models.
@mfroelund , @leifdenby and maintainers .. I request you to look up on this and guide me with your suggestions, once you approve , I will take enough time and implement things one by one.
The Problem
Currently,
mllam-verificationprovides an excellent foundation for continuous, point-wise verification (rmse,mae,difference). However, for high-resolution Limited Area Models (LAMs) like those developed inneural-lam, point-wise metrics often suffer from the well-known "double penalty" effect. If the model perfectly predicts a storm's intensity and structure but displaces it slightly, it is penalized twice (once for missing the actual location, and once for predicting it where it shouldn't be).Furthermore, as NWP shifts towards predicting extreme events, we currently lack threshold-based (categorical) verification metrics in the repository.
Proposed Solution
To make
mllam-verificationa complete diagnostic suite for high-resolution models, I propose adding a new suite of Spatial and Categorical verification metrics, leveragingxarrayand the existingcompute_pipeline_statisticinfrastructure.1. Spatial Verification: Fraction Skill Score (FSS)
Following Roberts and Lean (2008), FSS is the industry standard for evaluating high-resolution spatial forecasts. It compares the fractional coverage of events (e.g., precipitation > threshold) in spatial windows of increasing sizes.
fsswhich computes the score for a given threshold and neighborhood size (radius).plot_fss_scale()inplot.pyto visualize FSS as a function of neighborhood size, allowing researchers to determine the spatial scale at which a model becomes "useful" (typically FSS > 0.5).2. Categorical / Threshold Verification
Evaluate the model's ability to predict whether a variable exceeds a specific threshold (e.g., Wind Speed > 20m/s).
xarray-based generator for Hits, Misses, False Alarms, and Correct Negatives.Proposed Architecture & Integration
I plan to integrate these without breaking the current API:
mllam_verification/operations/statistics.pyor create dedicatedspatial.py/categorical.pymodules.xarray.DataArray.rolling(orscipy.signal.convolve2dwrapped viaxr.apply_ufunc) to compute fractional coverages efficiently across grid dimensions (x,y).plot.pywithplot_fss_scale(Line plot: Neighborhood Size on X-axis, FSS on Y-axis).Core Reason
As
neural-lambegins to integrate probabilistic models like Graph-EFM and CRPS-LAM, the ability to evaluate spatial structure and threshold exceedance becomes just as critical as continuous error. This contribution will provide the necessary tools for researchers to rigorously evaluate these next-generation models.@mfroelund , @leifdenby and maintainers .. I request you to look up on this and guide me with your suggestions, once you approve , I will take enough time and implement things one by one.