Update contribution templates and user guidance

[AnilKumar-NOAA]

Description:

Updates repository contribution and support guidance.

Changes include:

• Add GitHub issue forms and route enhancement requests to Discussions.
• Update PR template with labels, subcomponent checks, and affected areas.
• Clarify fork-and-PR workflow in contributing docs.
• Add official nested-EAGLE checkpoint download guidance.
• Add guidance for using a different dataset.
• Align README/support docs with issue vs discussion routing.

Checks:

• Parsed new issue template YAML files.
• Ran git diff --check.

Type of change:

• Bug fix
• New feature
• Refactor / cleanup
• Documentation
• CI/CD or tooling
• Other:

Area(s) affected

• nested_eagle workflow
• Verification / evaluation (via WXVX)
• Data prep / UFS2ARCO
• Config (YAML)
• Plotting / post-processing
• Infrastructure / Slurm scripts
• Other:

Commit Requirements:

• This PR addresses a relevant NOAA-EPIC/EAGLE issue (if not, create an issue); a person responsible for submitting the update has been assigned to the issue (link issue)
• Fill out all sections of this template.
• I have performed a self-review of my own code
• My changes generate no new warnings
• I have made corresponding changes to the system documentation if necessary

Testing / Verification:

• I ran and/or verified the changes (or provided a test plan)

• Commands/config used:

• Evidence (logs, key output paths, screenshots if relevant):

Runtime Environment:

• System/HPC:
• Account/role:
• Conda env:
• Key versions (optional):
  • python --version:
  • wxvx --version (if applicable):
  • MET version (if applicable):

Commit Message:

• UFS2ARCO -
• WXVX (verification) -

Subcomponent Pull Requests:

• EAGLE: NOAA-EPIC/EAGLE#
• UFS2ARCO: NOAA-PSL/UFS2ARCO#
• WXVX (verification): NOAA-EPIC/WXVX#
• None

[github-actions]

Link to ReadTheDocs sample build for this PR can be found at:
https://epic-eagle--194.org.readthedocs.build/en/194

[github-actions]

Link to ReadTheDocs sample build for this PR can be found at:
https://epic-eagle--194.org.readthedocs.build/en/194

[maddenp-cu]

There seems to be significant overlap between this section and the Type of change and Area(s) affected sections above. Has the submitter potentially already provided this information via one of those two sections? Would it be possible to somehow combine these three sections?

[github-actions]

Link to ReadTheDocs sample build for this PR can be found at:
https://epic-eagle--194.org.readthedocs.build/en/194

[maddenp-cu]

@AnilKumar-NOAA I'm sorry, I jumped into this PR too soon -- I see now that it's still marked Draft. I'll look again when you mark it Ready for review.

[AnilKumar-NOAA]

@AnilKumar-NOAA I'm sorry, I jumped into this PR too soon -- I see now that it's still marked Draft. I'll look again when you mark it Ready for review.

@maddenp-cu please keep checking and review internally as a team. We have marked draft right now but keep working on it before we provide for review to Josh and Priya (US team) and get the documentation up to standards approval from them.

[maddenp-cu]

Since there's a change to envs/, I think we should add the eagle-ursa label and let the nested-EAGLE workflow run. Hopefully I got the logic correct here.

[AnilKumar-NOAA]

here

Yes, I agree. Since this PR now changes envs/data.yaml, the relevance gate in nested_eagle.yml should pass because envs/ is included in the changed-file logic. The full Ursa pipeline is still intentionally label-gated, so an authorized maintainer needs to add the eagle-ursa label before it runs.

[github-actions]

Link to ReadTheDocs sample build for this PR can be found at:
https://epic-eagle--194.org.readthedocs.build/en/194

[AnilKumar-NOAA]

The failure was later during model initialization: the flash_attn CUDA extension installed in the Anemoi env is not compatible with the Torch/CUDA ABI on Ursa. I pushed a fix to use Anemoi’s PyTorch scaled_dot_product_attention backend in the EAGLE configs, which avoids importing the broken flash-attn extension.

[AnilKumar-NOAA]

@maddenp-cu The failure is different this time: CQC ran out of disk while building the hosted-runner environments, specifically while installing eagle-tools into the wxvx env. checking the existing filesystem-clean step to see whether we can free enough space before conda builds, or whether we should reduce what CQC builds.

[github-actions]

Link to ReadTheDocs sample build for this PR can be found at:
https://epic-eagle--194.org.readthedocs.build/en/194

[maddenp-cu]

the flash_attn CUDA extension installed in the Anemoi env is not compatible with the Torch/CUDA ABI on Ursa

Since when? This environment has been working correctly until today. What has changed?

I pushed a fix to use Anemoi’s PyTorch scaled_dot_product_attention backend in the EAGLE configs, which avoids importing the broken flash-attn extension.

But we want to use flash-attn, right? Is this a suggestion from an AI, or is this something that you researched and decided on and are recommending @AnilKumar-NOAA? What are the repercussions of this change?

[maddenp-cu]

@maddenp-cu The failure is different this time: CQC ran out of disk while building the hosted-runner environments, specifically while installing eagle-tools into the wxvx env. checking the existing filesystem-clean step to see whether we can free enough space before conda builds, or whether we should reduce what CQC builds.

This happens every now and then, and usually (in my experience so far) succeeds if re-run, as the next runner usually has more space available. I re-ran the failed job and will keep an eye on it. The new job already says 57 GB NEEDED, 89 GB AVAILABLE: OK, so that's promising.

EDIT: The failed job started with

57 GB NEEDED, 20 GB AVAILABLE: CLEANING

and finished with

RECLAIMED 38 GB
57 GB NEEDED, 58 GB AVAILABLE: OK

So maybe 57 GB is not sufficient given our current environments, which have been updated several times since this CQC workflow was created. I'm also wondering whether we can minimize the disk usage of the job by disabling or aggressively cleaning the conda and pip caches for CI jobs. I'll look into it sometime soon.

[AnilKumar-NOAA]

the flash_attn CUDA extension installed in the Anemoi env is not compatible with the Torch/CUDA ABI on Ursa

Since when? This environment has been working correctly until today. What has changed?

I pushed a fix to use Anemoi’s PyTorch scaled_dot_product_attention backend in the EAGLE configs, which avoids importing the broken flash-attn extension.

But we want to use flash-attn, right? Is this a suggestion from an AI, or is this something that you researched and decided on and are recommending @AnilKumar-NOAA? What are the repercussions of this change?

@maddenp-cu my PR is just documents update and I just saw this issue. I did some google search "the error message" "and also AI and just how to handle broken packages, and seems @DavidBurrows-NCO also facing this issue earlier- same error. Is this because of anemoi version change?

[AnilKumar-NOAA]

the flash_attn CUDA extension installed in the Anemoi env is not compatible with the Torch/CUDA ABI on Ursa

Since when? This environment has been working correctly until today. What has changed?

I pushed a fix to use Anemoi’s PyTorch scaled_dot_product_attention backend in the EAGLE configs, which avoids importing the broken flash-attn extension.

But we want to use flash-attn, right? Is this a suggestion from an AI, or is this something that you researched and decided on and are recommending @AnilKumar-NOAA? What are the repercussions of this change?

@maddenp-cu my PR is just documents update and I just saw this issue. I did some google search "the error message" "and also AI and just how to handle broken packages, and seems @DavidBurrows-NCO also facing this issue earlier- same error. Is this because of anemoi version change?

@maddenp-cu I can revert the SDPA config change and instead focus on the real environment fix: pin or adjust the compatible flash-attn/Torch/CUDA package set for Ursa.

[github-actions]

Link to ReadTheDocs sample build for this PR can be found at:
https://epic-eagle--194.org.readthedocs.build/en/194

[jdkublnick]

We have two proposed FAQs that we would like to add to the EAGLE User Guide prior to the capability announcement. Could we get someone to perform a technical review of the questions and answers below to ensure they are accurate before we move forward with publishing them?

Could we get someone on the EAGLE team to take a look? @DavidBurrows-NCO, @maddenp-cu, @AnilKumar-NOAA

Question 1:
Is there an easy way to generate the Anemoi training plots only for the nested HRRR domain, excluding the surrounding global domain?

Proposed Answer:
Yes. Users can define multiple bounding boxes to specify subset regions (e.g., CONUS, Europe, etc.) in the EAGLE pipeline base.yaml through the focus_areas option to generate plots that exclude the global domain. For example, for the HRRR/CONUS domain, you may use latlon_bbox: [22.0, -135.0, 50.0, -60.0] (or adjust as needed). Once added, the diagnostics will automatically generate plots restricted to these regions. For more information, users are advised to refer to the plot adaptor compatibility section of the Anemoi Training Diagnostic Documentation.

Question 2:
Is it feasible with the current nested EAGLE pipeline to add an HRRR variable (e.g., reflectivity) to the HRRR Zarr dataset even though it is not a GFS variable?

Proposed Answer:
No. The current nested EAGLE pipeline helps with interpolation and missing values, but not with adding domain-specific variables during nested training. Therefore, it does not support adding HRRR-only variables to a nested HRRR-over-GFS training setup. Compatibility checks ensure variables match between HRRR and GFS domains. To include reflectivity, users would need to start with a CONUS-only model, or find a way to create a placeholder for reflectivity or a derived field for the global domain, but this would require changes to the anemoi-training source code.

@AnilKumar-NOAA , once the technical review is complete by your team, could you please add these FAQs to the User Guide under docs/reference/faq.rst

[jdkublnick]

In quickstart.rst, could we add repository cloning instructions near the beginning? The Quickstart guide should be all-encompassing, but currently assumes users already have the repository cloned and are in the repository root directory.

[jdkublnick]

A few suggestions for the nested_eagle.rst lines 79-81:

• Add a code block under step 1 showing how to check out the nested-eagle-v1 branch (e.g., git checkout nested-eagle-v1) to keep it consistent with the rest of the Quickstart.
• Update line 80 to: " Follow the NRT workflow in :ref:NRT "
• Add an NRT label (e.g., .. _NRT:) above the "Run inference in near-real-time (NRT)" section in quickstart.rst so the reference resolves to the correct location.

[jdkublnick]

Please add the following terms to the glossary:

AORC
Analysis of Record for Calibration. A NOAA gridded meteorological forcing dataset used for land surface and hydrologic applications.

checkpoint
A saved snapshot of a trained machine learning model that can be used to resume training or perform inference.

CONUS
Continental United States.

ERA5
The fifth-generation atmospheric reanalysis produced by the European Centre for Medium-Range Weather Forecasts (ECMWF).

GEFS
Global Ensemble Forecast System. NOAA's global ensemble weather prediction system.

MET
Model Evaluation Tools. A community-developed verification package used to evaluate numerical weather prediction and machine learning weather forecasts.

netCDF
Network Common Data Form. A file format commonly used for storing and sharing scientific array-based data.

NRT
Near real-time. Refers to workflows that generate forecasts using the latest available input data shortly after it becomes available.

PrepBUFR
A BUFR-formatted observational dataset used for numerical weather prediction and forecast verification.

[github-actions]

Link to ReadTheDocs sample build for this PR can be found at:
https://epic-eagle--194.org.readthedocs.build/en/194

[maddenp-cu]

Question 1: Is there an easy way to generate the Anemoi training plots only for the nested HRRR domain, excluding the surrounding global domain?

Proposed Answer: Yes. Users can define multiple bounding boxes to specify subset regions (e.g., CONUS, Europe, etc.) in the EAGLE pipeline base.yaml through the focus_areas option to generate plots that exclude the global domain. For example, for the HRRR/CONUS domain, you may use latlon_bbox: [22.0, -135.0, 50.0, -60.0] (or adjust as needed). Once added, the diagnostics will automatically generate plots restricted to these regions. For more information, users are advised to refer to the plot adaptor compatibility section of the Anemoi Training Diagnostic Documentation.

@jdkublnick I don't know Anemoi well enough to weigh in on the correctness of Answer 1, but I have a couple of tangential comments:

1. Your suggestion gives a link to a specific line in a specific commit of base.yaml. When I click that, I'm not sure I understand what I'm supposed to change. The highlighted line? Actually, I think I understand from the linked-to Anemoi docs that I'm meant to insert a focus_area key-value pair here. If so, I wonder if it might help to update the link to highlight the entire plotting config block like this, maybe advising something like "A focus_area key can be added to this config block".
2. My understanding is that Anemoi is always evolving, and base.yaml will probably evolve in sync with it, so a link to a specific hash might become out-of-date. Referring more obliquely to the value to change via a YAML key path, like "The PlotSample element in the training.anemoi.diagnostics.plot.callbacks sequence" might be more stable, though even that might change. Just a thought, not really a recommendation; there's no real alternative to reviewing documentation for accuracy.
3. Users can change base.yaml directly, but they might be better advised to make their changes in the eagle.yaml that they create in Step 1.1.1 of the Quickstart Guide, as a change made directly to base.yaml could be overwritten during config composition. Obviously we wouldn't be able to give a hyperlink to eagle.yaml as it's not a git-controlled file, so a link to base.yaml might still be useful for orientation, but a change made to the user's eagle.yaml will be the safest.

@AnilKumar-NOAA I just noticed that we have two 1. blocks in this section -- is that something you could fix in this PR?

[maddenp-cu]

@AnilKumar-NOAA I just noticed that we have two 1. blocks in this section -- is that something you could fix in this PR?

@AnilKumar-NOAA I think it might be because this note needs to be indented, and is breaking the auto-numbering because it is not.

[github-actions]

Link to ReadTheDocs sample build for this PR can be found at:
https://epic-eagle--194.org.readthedocs.build/en/194

[AnilKumar-NOAA]

@AnilKumar-NOAA I just noticed that we have two 1. blocks in this section -- is that something you could fix in this PR?

@AnilKumar-NOAA I think it might be because this note needs to be indented, and is breaking the auto-numbering because it is not.

@maddenp-cu Thanks, fixed the Quickstart numbering issue by indenting the note under step 1 and keeping the NRT label inside the list item.

[jdkublnick]

@AnilKumar-NOAA, please update FAQ 1, then I believe this PR will be ready to merge. Thank you!

Question 1:
Is there an easy way to generate the Anemoi training plots only for the nested HRRR domain, excluding the surrounding global domain?

Proposed Answer:

Yes. Users can define multiple bounding boxes to specify subset regions (e.g., CONUS, Europe, etc.) by adding a focus_areas key to the PlotSample configuration block in the EAGLE pipeline base.yaml. This configuration generates plots that exclude the surrounding global domain.

Although users can modify base.yaml directly for reference, it is recommended that they instead add the configuration to the eagle.yaml file created during Step 1.1.1 of the Quickstart Guide. Changes made directly to base.yaml may be overwritten during configuration composition, whereas updates made in eagle.yaml are preserved.

For example, for the HRRR/CONUS domain, you may use latlon_bbox: [22.0, -135.0, 50.0, -60.0] (or adjust as needed). Once added, the diagnostics will automatically generate plots restricted to these regions. For more information, users are advised to refer to the plot adaptor compatibility section of the Anemoi Training Diagnostic Documentation.

[github-actions]

Link to ReadTheDocs sample build for this PR can be found at:
https://epic-eagle--194.org.readthedocs.build/en/194