Switch to Using ClawpackTestRunner for Testing by mandli · Pull Request #697 · clawpack/geoclaw

mandli · 2026-03-17T14:29:40Z

This PR aims to use the new testing framework now used in AMRClaw and Classic to run regression tests from clawutil's ClawpackTestRunner class. This switches everything over to using PyTest and as much functionality as possible from there.

Major goals for this particular PR is to use the same testing structure as was in AMRClaw and Classic, but change things minimally to get things working before moving onto to larger structural changes.

Also adds marks for test groupings and default set of tests to run. This also adds new regression test data for storm surge as the test has been changed to be more robust and cover more of the package. This also includes depending on a local storm file rather than attempting to download data (covered elsewhere in a Python unit test).

The plotting tests have been split into "smoke" tests (does it raise an exception) and actual plotting tests that can be run on request.

…resent

Included some bug fixes in storm module for formats that are tested but not used very often. Need to come back to fix the IBTrACS test.

Note: test_dtopo_io has been marked as expected to fail with the measured 5e-4 discrepency noted. This may be due to the precision being used by default to write out the files but needs follow up.

Provides seperation of structure (plotting and writing of data) similar to how other topotools tests are carried out.

Also simplifies flagging for building the multi-layer code against LAPACK.

Also fixes some things in the testing.yml file so that it correctly archives test output on failure and gives more output for debugging. Also adds so that tests that use it can be tested.

These were added, but two are currently broken as the read functions do not work. These have been marked as expected to fail. The SurgeData test now works, but did require a bug fix.

This test is failing due to what appears to be an off by one error and just a bit of a mismatch in the regression data. This requires more looking into, but the test works.

Does not match gauge output yet and needs a validation pass to look at gauge changes.

mandli · 2026-04-03T14:40:10Z

Phew, all the tests from before are now implemented! There are a few issues that need to be looked at though:

Tests marked as xfail/xpass:
- tests/test_storm.py::test_storm_io[ibtracs] - This is expected as ibtracs have not been fully implemented
- tests/test_dtopotools.py::test_dtopo_io - Failing occasionally, need to look at read/write precision
- tests/test_data.py::test_read_fgmax_data_multiple_grids - FGmaxData.read does not yet round-trip this multi-grid/point_style case correctly.
  correctly.
- tests/test_data.py::test_dtopo_data_roundtrip - DTopoData.read currently fails to parse this written dtopo.data round-trip case.
- examples/tsunami/island-particles/test_particles.py - Slight gauge mismatch
- examples/tsunami/dtopo/test_dtopo.py::test_dtopo - Slight gauge mismatch
- examples/tsunami/chile2010_adjoint/test_chile2010_adjoint_forward.py - Slight gauge mismatch
Skipped
- tests/test_dtopotools.py::test_geometry - Uses old tools and may be duplicative with use of newer tools
- tests/test_dtopotools.py - Skipped comparison with old tools

More test coverage is possible, but if we can merge this in, we can also systematically correct the remaining regression data issues.

mandli · 2026-04-03T14:41:34Z

Current markers implemented (in pytest.ini):

Test type (choose one):
- python: pure Python unit tests
- regression: example-based regression tests (Fortran)
Execution characteristics:
- slow: slower tests not intended for the default quick CI path
- remote: tests that require fetching a file from the web
- netcdf: tests requiring NetCDF support
Domain / feature grouping:
- tsunami: tsunami related tests
- storm: storm surge related tests
- multilayer: multilayer shallow water related tests
- adjoint: inverse/adjoint solver releated tests
- adjoint_forward: forward runs that use inverse/adjoint solver, uses adjoint data

mandli · 2026-04-03T17:39:10Z

I have now split the testing into three groups:

python unit testing, which will eventually also include linting.
regression testing
slow regression testing

There's really only one slow regression test right now, and it really does not do anything but run a longer run. The intent is that the slow regression testing will happen periodically (daily at 6 AM UTC right now). We could also add another one that would do the gallery examples, or do the Jupyter notebooks associated with the repos. Wanted to get an example running though so that we can think about how to use it.

If this looks good, I can move this structure to AMRClaw and classic when it makes sense.

rjleveque · 2026-04-04T20:44:36Z

Merging this along with clawpack/classic#96 and clawpack/amrclaw#310.

mandli added 6 commits March 17, 2026 10:19

Convert test module to new clawpack test runner

cca9dea

Convert test_units to pytest

e3809fd

Add new testing matrix for GitHub CI

96120fd

Select only working tesets and update action versions

a9f359c

Temporarily disable linting

a2e3976

Update bowl-slosh

4f7d6bb

mandli force-pushed the switch-test-runner branch from f7ba443 to 4f7d6bb Compare March 17, 2026 15:27

mandli added 23 commits March 18, 2026 11:25

Remove gauge 1 from storm regression test

04b1468

Handle finer case for ike test

cb41daf

Correct CI testing path

14877b8

Add pandas dependency to CI

b1d7b4e

Convert over test_util.py

74f7ac2

Add topo_topotools to new testing framework

6e0fc81

The plotting tests have been split into "smoke" tests (does it raise an exception) and actual plotting tests that can be run on request.

Remove plot marker, rely on skipping of test when matplotlib is not p…

ebd1595

…resent

Rework unstructured, remote and netcdf tests

9329cd9

Convert over storm tests

e68af2c

Included some bug fixes in storm module for formats that are tested but not used very often. Need to come back to fix the IBTrACS test.

Fix up unit tests

6cb61c8

Convert test_dtopotools

cfe84cd

Note: test_dtopo_io has been marked as expected to fail with the measured 5e-4 discrepency noted. This may be due to the precision being used by default to write out the files but needs follow up.

Merge test_etopo1 into test_topotools

aa509b1

Provides seperation of structure (plotting and writing of data) similar to how other topotools tests are carried out.

Move tests from isaac to test_storm

6212472

Fix categorization bug and add tests

0512d85

Update multi-layer plane_wave test regression data

811f719

Also simplifies flagging for building the multi-layer code against LAPACK.

Add LAPACK as a dependency for multi-layer testing

b779bb4

Also fixes some things in the testing.yml file so that it correctly archives test output on failure and gives more output for debugging. Also adds so that tests that use it can be tested.

Add netcdf4 python package to CI

fefc57d

Remove old regression tests

2e02a73

Add more data module tests

7569354

These were added, but two are currently broken as the read functions do not work. These have been marked as expected to fail. The SurgeData test now works, but did require a bug fix.

Convert over dtopo test

339992d

This test is failing due to what appears to be an off by one error and just a bit of a mismatch in the regression data. This requires more looking into, but the test works.

Convert particles test

aa07545

Does not match gauge output yet and needs a validation pass to look at gauge changes.

Correct testing.yml path to test

daf03f7

mandli mentioned this pull request Apr 2, 2026

New pytest test runner clawpack/amrclaw#310

Merged

mandli added 13 commits April 2, 2026 19:07

Fix pointing to load_module

41c9e93

Fix another problematic load_module

5073858

Switch to fullpath_module

95a8b5a

Fix archiving for CI

6c7d693

Enable verbose reporting in test CI

372ee11

Remove debugging and reenable all tests

b019378

Turn on debugging for CI

c551a9d

Try enabling more tests for CI

259e0dc

Turn back on all python tests

3f552da

Add fix for tests breaking with write_geoclaw

60644c9

Turn back on all tests

44943eb

Remove leftover old adjoint test

7474f71

Update test readme and make minor tests adjustments

3ab25da

mandli added 6 commits April 3, 2026 10:42

Tweak markers and testing steps

0615165

Rework CI testing

716190d

Add slow tests GitHub action

c41f467

Add missing flagging script for CI

d329fcd

Add compiler support back to python

fa5ae8c

Correct path inconsistencies in CI actions

ed37878

mandli added 5 commits April 3, 2026 13:46

Setup Ike test for slow regression comparison

309ede4

Standarize CLI calling of tests

f8c1a1a

Missed a couple

702896c

Add missing save in tests

dd61442

Correct some incorrect saves

15ecc0d

rjleveque merged commit 7ef5416 into clawpack:master Apr 4, 2026
8 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Switch to Using ClawpackTestRunner for Testing#697

Switch to Using ClawpackTestRunner for Testing#697
rjleveque merged 70 commits into
clawpack:masterfrom
mandli:switch-test-runner

mandli commented Mar 17, 2026

Uh oh!

mandli commented Apr 3, 2026 •

edited

Loading

Uh oh!

mandli commented Apr 3, 2026

Uh oh!

mandli commented Apr 3, 2026 •

edited

Loading

Uh oh!

rjleveque commented Apr 4, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

mandli commented Mar 17, 2026

Uh oh!

mandli commented Apr 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mandli commented Apr 3, 2026

Uh oh!

mandli commented Apr 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rjleveque commented Apr 4, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mandli commented Apr 3, 2026 •

edited

Loading

mandli commented Apr 3, 2026 •

edited

Loading