Skip to content

test: register slow pytest marker and apply to training tests#651

Merged
sadamov merged 1 commit into
mllam:mainfrom
sadamov:test/run-slow-marker
Jun 9, 2026
Merged

test: register slow pytest marker and apply to training tests#651
sadamov merged 1 commit into
mllam:mainfrom
sadamov:test/run-slow-marker

Conversation

@sadamov

@sadamov sadamov commented Jun 7, 2026

Copy link
Copy Markdown
Collaborator

Describe your changes

Register a slow pytest marker in pyproject.toml and apply @pytest.mark.slow to test_training (parametrised over all datastores) and test_training_output_std because they run a real trainer.fit loop and take minutes per parametrisation.

This relies on pytest's built-in marker filtering rather than introducing a custom flag:

  • pytest (no flags) runs the full suite, slow tests included. CI uses this and continues to exercise everything.
  • pytest -m "not slow" skips slow tests, for fast local iteration during contributor development.
  • pytest -m slow runs only slow tests, e.g. when debugging a training regression.

Motivation: I needed this in #635 for the ERA5 boundary integration test, then realised the mechanism is independently useful for the existing trainer.fit-based tests too. Landing the marker separately so #635 doesn't have to ship two things at once, and so #231's test_state_only_datastore_training_setup_runs (and any future trainer.fit-based regression) can opt in via the same marker.

Coordination with #615 (@jishanahmed-shaikh): orthogonal. The slow marker is about execution time, the unit/integration split is about scope. They coexist: an integration test can be fast (DummyDatastore) and a unit test can be slow (large numerical fixture).

Earlier iteration of this PR introduced a custom --run-slow CLI flag with skip-by-default behaviour. @observingClouds suggested the inverse (run-by-default with --skip-slow), and the cleanest resolution was to drop the custom flag entirely and rely on pytest's native -m selection. Pytest already supports exactly this pattern with zero custom code.

No new dependencies. No CI changes.

Verified locally:

  • pytest tests/test_training.py --collect-only -q -> all 6 tests collected
  • pytest tests/test_training.py -m "not slow" --collect-only -q -> 2 collected, 4 deselected

Issue Link

Refs #635, refs #599

Type of change

  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • ✨ New feature (non-breaking change that adds functionality)
  • 💥 Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • 📖 Documentation (Addition or improvements to documentation)

Checklist before requesting a review

  • My branch is up-to-date with the target branch - if not update your fork with the changes from the target branch (use pull with --rebase option if possible).
  • I have performed a self-review of my code
  • For any new/modified functions/classes I have added docstrings that clearly describe its purpose, expected inputs and returned values
  • I have placed in-line comments to clarify the intent of any hard-to-understand passages of my code
  • I have updated the README to cover introduced code changes
  • I have added tests that prove my fix is effective or that my feature works
  • I have given the PR a name that clearly describes the change, written in imperative form (context).
  • I have requested a reviewer and an assignee (assignee is responsible for merging). This applies only if you have write access to the repo, otherwise feel free to tag a maintainer to add a reviewer and assignee.

Checklist for reviewers

Each PR comes with its own improvements and flaws. The reviewer should check the following:

  • the code is readable
  • the code is well tested
  • the code is documented (including return types and parameters)
  • the code is easy to maintain

Author checklist after completed review

  • I have added a line to the CHANGELOG describing this change, in a section
    reflecting type of change (add section where missing):
    • added: when you have added new functionality
    • changed: when default behaviour of the code has been changed
    • fixes: when your contribution fixes a bug
    • maintenance: when your contribution is relates to repo maintenance, e.g. CI/CD or documentation

Checklist for assignee

  • PR is up to date with the base branch
  • the tests pass
  • (if the PR is not just maintenance/bugfix) the PR is assigned to the next milestone. If it is not, propose it for a future milestone.
  • author has added an entry to the changelog (and designated the change as added, changed, fixed or maintenance)
  • Once the PR is ready to be merged, squash commits and merge the PR.

@sadamov sadamov added cicd maintenance Refactoring codebase, no new behaviour labels Jun 7, 2026
@sadamov sadamov requested a review from observingClouds June 7, 2026 14:18
@sadamov sadamov self-assigned this Jun 7, 2026
sadamov added a commit to sadamov/neural-lam that referenced this pull request Jun 7, 2026
Matches the CI invocation introduced in mllam#651 so this branch's eventual
ERA5 boundary integration test (and any other @pytest.mark.slow on this
branch) is exercised by CI, not just by local --run-slow invocations.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
sadamov added a commit to sadamov/neural-lam that referenced this pull request Jun 7, 2026
Matches the CI invocation introduced in mllam#651 so any @pytest.mark.slow
test on this branch is exercised by CI, not just by local --run-slow
invocations.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

@observingClouds observingClouds left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a great idea @sadamov. I wonder if it is better to define the inverse of this, e.g. a --skip-slow flag that needs to be set. So by default the whole suite is run and only in the case when you are familiar with the testing, you add the --skip-slow flag to skip some tests.

sadamov added a commit to sadamov/neural-lam that referenced this pull request Jun 8, 2026
Per @observingClouds's review on mllam#651: pytest already has built-in
marker filtering, so we don't need the custom pytest_addoption +
pytest_collection_modifyitems hook to support skip-by-default
behaviour. Drop the custom flag entirely and rely on pytest's
native -m marker selection:

- `pytest` (no flags) runs the full suite, including slow tests
  (this restores the prior default behaviour, addresses observingClouds'
  safety concern).
- `pytest -m "not slow"` skips slow tests for fast local iteration.
- CI runs `pytest` unchanged - no --run-slow flag needed.

Keep only the marker registration in pyproject.toml and the
@pytest.mark.slow decorators on test_training and
test_training_output_std.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@sadamov sadamov changed the title test: introduce --run-slow pytest flag and mark training tests as slow test: register slow pytest marker and apply to training tests Jun 8, 2026
@sadamov

sadamov commented Jun 8, 2026

Copy link
Copy Markdown
Collaborator Author

@observingClouds, I am glad you mentioned this. I just learned that there is a built-in -m marker selection in pytest 35329d9. Default pytest now runs the full suite (slow tests included), and contributors can opt out with pytest -m "not slow" during local iteration. CI is also back to plain pytest with no flag.
This should be exactly what you requested and the built-in reduced the number of lines to 9 :D

sadamov added a commit to sadamov/neural-lam that referenced this pull request Jun 8, 2026
Per the simplification in mllam#651, the custom --run-slow flag is being
dropped in favour of relying on pytest's native -m marker selection.
Remove the corresponding pytest_addoption + pytest_collection_modifyitems
from conftest and the --run-slow flag from CI on this branch too, so
this PR doesn't re-introduce conflicts when mllam#651 lands. The slow marker
registration in pyproject.toml stays, ready for use on any future
@pytest.mark.slow test on this branch.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
sadamov added a commit to sadamov/neural-lam that referenced this pull request Jun 8, 2026
Per the simplification in mllam#651, the custom --run-slow flag is being
dropped in favour of relying on pytest's native -m marker selection.
Remove the corresponding pytest_addoption + pytest_collection_modifyitems
from conftest and the --run-slow flag from CI on this branch too, so
this PR doesn't re-introduce conflicts when mllam#651 lands.

The @pytest.mark.slow on test_plot_prediction_with_era5_boundary stays;
under the new mechanism that test runs by default (including in CI) and
contributors can skip it locally with `pytest -m "not slow"`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@sadamov sadamov requested a review from observingClouds June 8, 2026 18:32
sadamov added a commit to ANANYA542/neural-lam that referenced this pull request Jun 8, 2026
CONTRIBUTING.md (this PR) now owns the general contributor workflow.
Remove the duplicate sections from AGENTS.md (Issues, Pull Requests,
Changelog) and add a pointer to CONTRIBUTING.md as the first thing
agents should read.

Kept and refined the AI-specific rules:
- Search before creating issues/PRs (now consolidated to one section
  with the exact gh search commands).
- Re-read the full thread before every action, reload after context
  gaps - the rule that matters most for agents.
- Communication style (terse, no filler ban list).
- AI attribution in commit trailers.

Also updated the codebase reference to reflect the post-mllam#208
ForecasterModule / Forecaster / StepPredictor hierarchy, and added
`pytest -m "not slow"` to the command list (per mllam#651).

Net: 91 lines (down from 97), but the AI-specific content is denser.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Comment thread CHANGELOG.md Outdated
Comment thread pyproject.toml Outdated

@observingClouds observingClouds left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a few more small suggestions.

sadamov added a commit to sadamov/neural-lam that referenced this pull request Jun 9, 2026
Match the marker registration used on mllam#635/mllam#651 so any future
@pytest.mark.slow test on this branch is recognised by pytest
without warnings. No tests currently use the marker on mllam#656.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

This comment was marked as spam.

Copilot AI mentioned this pull request Jun 9, 2026
6 tasks
Add a `[tool.pytest.ini_options]` block to `pyproject.toml` that
registers a `slow` marker, and apply `@pytest.mark.slow` to
`test_training` and `test_training_output_std` so contributors can
skip the long training tests during local iteration:

    pytest -m "not slow"  # quick sweep, skips training tests
    pytest -m slow        # only the training tests
    pytest                # everything (CI default)

Closes mllam#651.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@sadamov sadamov force-pushed the test/run-slow-marker branch from a7e7d68 to 6dd6c07 Compare June 9, 2026 08:38
@sadamov sadamov merged commit b668a58 into mllam:main Jun 9, 2026
11 checks passed
@sadamov sadamov deleted the test/run-slow-marker branch June 9, 2026 09:46
sadamov added a commit to Jayant-kernel/neural-lam-gsoc- that referenced this pull request Jun 9, 2026
`test_last_checkpoint_saved_without_validation` spins up a real
`pl.Trainer` and runs a full training epoch on CPU; in practice this
adds ~60s to a pytest run, same order as the `test_training` tests
already marked `slow` in mllam#651. Tagging it here so it is grouped with
the other long-running training tests and skipped by default during
fast `pytest -m "not slow"` iteration.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
sadamov added a commit to sadamov/neural-lam that referenced this pull request Jun 16, 2026
The `slow` marker registration still described the markers as "deselected by
default" (a leftover from the removed `--run-slow` flag). Slow tests actually
run by default and are skipped with `-m "not slow"`, matching the behaviour
agreed in mllam#651. Description-only change; no behaviour change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cicd maintenance Refactoring codebase, no new behaviour

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants