Running pytest coverage for a specific dataset causes an error

[calebrob6]

Not sure if this is torchgeo or pytest-cov or torchvision specific but recording it here.

Our docs show an example of measuring test coverage for a specific file: pytest --cov=torchgeo.datasets.foo tests/datasets/test_foo.py, however if I run this (e.g. pytest --cov=torchgeo.datasets.landcoverai tests/datasets/test_landcoverai.py then I get the following ImportError:

ImportError while loading conftest '/home/caleb/torchgeo/tests/datasets/conftest.py'.
tests/datasets/conftest.py:9: in <module>
    import torchvision.datasets.utils
../anaconda3/envs/torchgeo/lib/python3.12/site-packages/torchvision/__init__.py:10: in <module>
    from torchvision import _meta_registrations, datasets, io, models, ops, transforms, utils  # usort:skip
../anaconda3/envs/torchgeo/lib/python3.12/site-packages/torchvision/_meta_registrations.py:25: in <module>
    @register_meta("roi_align")
../anaconda3/envs/torchgeo/lib/python3.12/site-packages/torchvision/_meta_registrations.py:19: in wrapper
    get_meta_lib().impl(getattr(getattr(torch.ops.torchvision, op_name), overload_name), fn)
../anaconda3/envs/torchgeo/lib/python3.12/site-packages/torch/library.py:296: in impl
    raise RuntimeError(
E   RuntimeError: This is not allowed since there's already a kernel registered from python overriding roi_align's behavior for Meta dispatch key and torchvision namespace.

Weirdly, pytest --cov=torchgeo.datasets tests/datasets/test_landcoverai.py works just fine.

Also, I'll note that the initial command definitely worked at some point in time, and it works with a decently old environment that had torchvision 0.15 installed.

Shout out to @amritagupta for the initial report, thanks!

[adamjstewart]

I get a completely different set of errors:

FAILED tests/datasets/test_landcoverai.py::TestLandCoverAIGeo::test_already_downloaded - rasterio.errors.PathError: invalid path '_UnparsedPath(path='/private/var/folders/jv/cgkfvslj6nq1l7cw0c8c_8gm0000gn/T/pytest-of-Adam/pytest-9/test_already_downloaded0/images/M-33-20-D-c-4-2.tif')'
ERROR tests/datasets/test_landcoverai.py::TestLandCoverAIGeo::test_getitem - rasterio.errors.PathError: invalid path '_UnparsedPath(path='/private/var/folders/jv/cgkfvslj6nq1l7cw0c8c_8gm0000gn/T/pytest-of-Adam/pytest-9/test_getitem0/images/M-33-20-D-c-4-2.tif')'
ERROR tests/datasets/test_landcoverai.py::TestLandCoverAIGeo::test_already_extracted - rasterio.errors.PathError: invalid path '_UnparsedPath(path='/private/var/folders/jv/cgkfvslj6nq1l7cw0c8c_8gm0000gn/T/pytest-of-Adam/pytest-9/test_already_extracted0/images/M-33-20-D-c-4-2.tif')'
ERROR tests/datasets/test_landcoverai.py::TestLandCoverAIGeo::test_out_of_bounds_query - rasterio.errors.PathError: invalid path '_UnparsedPath(path='/private/var/folders/jv/cgkfvslj6nq1l7cw0c8c_8gm0000gn/T/pytest-of-Adam/pytest-9/test_out_of_bounds_query0/images/M-33-20-D-c-4-2.tif')'
ERROR tests/datasets/test_landcoverai.py::TestLandCoverAIGeo::test_plot - rasterio.errors.PathError: invalid path '_UnparsedPath(path='/private/var/folders/jv/cgkfvslj6nq1l7cw0c8c_8gm0000gn/T/pytest-of-Adam/pytest-9/test_plot0/images/M-33-20-D-c-4-2.tif')'

I think we need to better understand the import logic introduced by using --cov and figure out where it's causing issues.

[calebrob6]

Wow! This is with the same versions of packages used by our CI?

[adamjstewart]

Probably slightly different versions. To make matters more interesting, if I convert the pytest-cov logic to coverage.py, I get a third error:

> coverage run --source=torchgeo.datasets.landcoverai -m pytest tests/datasets/test_landcoverai.py
ERROR: while parsing the following warning configuration:

  ignore:The .*dataloader.* does not have many workers which may be a bottleneck:lightning.fabric.utilities.warnings.PossibleUserWarning:lightning

This error occurred:

Traceback (most recent call last):
  File "/Users/Adam/spack/var/spack/environments/default/.spack-env/view/lib/python3.13/site-packages/_pytest/config/__init__.py", line 1921, in parse_warning_filter
    category: Type[Warning] = _resolve_warning_category(category_)
                              ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/Users/Adam/spack/var/spack/environments/default/.spack-env/view/lib/python3.13/site-packages/_pytest/config/__init__.py", line 1959, in _resolve_warning_category
    m = __import__(module, None, None, [klass])
  File "/Users/Adam/spack/var/spack/environments/default/.spack-env/view/lib/python3.13/site-packages/lightning/__init__.py", line 18, in <module>
    from lightning.fabric.fabric import Fabric  # noqa: E402
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/Adam/spack/var/spack/environments/default/.spack-env/view/lib/python3.13/site-packages/lightning/fabric/__init__.py", line 30, in <module>
    from lightning.fabric.fabric import Fabric  # noqa: E402
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/Adam/spack/var/spack/environments/default/.spack-env/view/lib/python3.13/site-packages/lightning/fabric/fabric.py", line 35, in <module>
    import torch
  File "/Users/Adam/spack/var/spack/environments/default/.spack-env/view/lib/python3.13/site-packages/torch/__init__.py", line 1703, in <module>
    from torch._tensor import Tensor  # usort: skip
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/Adam/spack/var/spack/environments/default/.spack-env/view/lib/python3.13/site-packages/torch/_tensor.py", line 21, in <module>
    from torch.overrides import (
    ...<5 lines>...
    )
  File "/Users/Adam/spack/var/spack/environments/default/.spack-env/view/lib/python3.13/site-packages/torch/overrides.py", line 1754, in <module>
    has_torch_function = _add_docstr(
        _has_torch_function,
    ...<18 lines>...
        """,
    )
RuntimeError: function '_has_torch_function' already has a docstring


/Users/Adam/spack/var/spack/environments/default/.spack-env/view/lib/python3.13/site-packages/coverage/inorout.py:507: CoverageWarning: Module torchgeo.datasets.landcoverai was never imported. (module-not-imported)
  self.warn(f"Module {pkg} was never imported.", slug="module-not-imported")
/Users/Adam/spack/var/spack/environments/default/.spack-env/view/lib/python3.13/site-packages/coverage/control.py:860: CoverageWarning: No data was collected. (no-data-collected)
  self._warn("No data was collected.", slug="no-data-collected")

[adamjstewart]

Trying to start discussion at https://discuss.python.org/t/how-to-run-coverage-for-specific-file/46019/5 before opening an issue on GitHub since reproducing is complicated at best.

[adamjstewart]

Upgraded my environment with newer versions of everything and now I get the same error as you.

[adamjstewart]

nedbat/coveragepy#1925