Skip to content

Conversation

Bastian-Krause
Copy link
Member

@Bastian-Krause Bastian-Krause commented Sep 30, 2025

Description
@pytest.mark.lg_xfail_feature() is meant to mark a test as "xfail" if the target or global environment contains the specified feature flag. There is no other sane way of testing for features in pytest's collection phase. Marking a test as xfail later is unsupported in pytest. So a pytest.mark decorator is necessary.

Imagine two targets contain the camera feature flag. One of them has the additional special-camera-2000 feature flag. The other has the additional special-camera-3000 feature flag. Due to a known bug on special-camera-3000, the test is expected to fail. The test can be marked as xfail for that feature:

import pytest

@pytest.mark.lg_feature("camera"])
@pytest.mark.lg_xfail_feature(
    "special-camera-3000",
    reason="known bug xy on special-camera-3000",
    raises=AssertionError,
    strict=True,
)
def test_camera(target):
    pass

While at it, improve error messages, skip reason and documentation of @pytest.mark.lg_feature() and feature flags. Especially the documentation changes make it easier to add documentation for the new @pytest.mark.lg_xfail_feature().

Checklist

  • Documentation for the feature
  • Tests for the feature
  • Add a section on how to use the feature to doc/usage.rst
  • PR has been tested

Copy link

codecov bot commented Sep 30, 2025

Codecov Report

❌ Patch coverage is 0% with 23 lines in your changes missing coverage. Please review.
✅ Project coverage is 45.3%. Comparing base (2daa13d) to head (edee111).
⚠️ Report is 2 commits behind head on master.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
labgrid/pytestplugin/hooks.py 0.0% 23 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff            @@
##           master   #1751     +/-   ##
========================================
- Coverage    45.4%   45.3%   -0.1%     
========================================
  Files         172     172             
  Lines       13503   13516     +13     
========================================
  Hits         6131    6131             
- Misses       7372    7385     +13     
Flag Coverage Δ
3.10 45.3% <0.0%> (-0.1%) ⬇️
3.11 45.3% <0.0%> (-0.1%) ⬇️
3.12 45.3% <0.0%> (-0.1%) ⬇️
3.13 45.3% <0.0%> (-0.1%) ⬇️
3.9 45.4% <0.0%> (-0.1%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@Emantor Emantor self-requested a review October 1, 2025 09:54
Copy link
Member

@Emantor Emantor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good, some smaller suggestions.

…d error messages

Explain the purpose of the marker, turn the bare Exception into a
pytest.UsageError, improve and simplify the error message and skip reason.

Signed-off-by: Bastian Krause <[email protected]>
Every test creates a bare env config with or without feature. A future
commit will introduce even more tests that will do this.

So move the env config creation to a factory fixture that takes the
desired features as an argument.

Signed-off-by: Bastian Krause <[email protected]>
Tests if the appropriate pytest.UsageErrors are raised for calls to
@pytest.mark.lg_feature() with unexpected arguments.

Signed-off-by: Bastian Krause <[email protected]>
Adding more headlines makes it easier to scan over these sections when
searching for something. It also helps pointing people to these sections
due to the HTML anchors available for headlines in the HTML version of
the docs.

Signed-off-by: Bastian Krause <[email protected]>
Currently, the pytest plugin section lacks some structure:

- Command-Line Options
- Environment Variables
  - LG_ENV
  - LG_PROXY
- Simple Example
- Custom Fixture Example
- Strategy Fixture Example
- Feature Flags
- Test Reports

It's a mixture of tutorial-like examples, basic explanations and features.

Move the examples into their own section and make their headlines more
catchy.

Signed-off-by: Bastian Krause <[email protected]>
… docs

The feature flags are not a pytest plugin feature. Instead they should
be described in the configuration chapter in the environment config
section. So move them there.

The pytest plugin docs should, however, document
@pytest.mark.lg_feature, so name the section accordingly and simplify
the section, so the relevant parts are described there in a pointed
fashion.

Signed-off-by: Bastian Krause <[email protected]>
`@pytest.mark.lg_xfail_feature()` is meant to mark a test as "xfail" if
the target or global env contains the specified feature flag.

Signed-off-by: Bastian Krause <[email protected]>
@Bastian-Krause
Copy link
Member Author

Changes:

  • raise pytest.UsageError for unexpected marker args and add expected function signatures to error message (for both @pytest.mark.lg_feature and @pytest.mark.lg_xfail_feature)
  • add/use env_feature_config factory fixture for env config creation to tests/test_flags.py
  • add tests for pytest.UsageErrors (for both @pytest.mark.lg_feature and @pytest.mark.lg_xfail_feature)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants