Skip to content

Commit

Permalink
feat: add support for custom xfail statuses in pytest
Browse files Browse the repository at this point in the history 
- Support for custom statuses of xfail-marked tests in pytest.
- Default statuses: `skipped` (failed xfail) and `passed` (successful xfail).
- Configuration values can be set via `qase.config.json` or environment variables:
    - `QASE_PYTEST_XFAIL_STATUS_XFAIL`
    - `QASE_PYTEST_XFAIL_STATUS_XPASS`
  • Loading branch information
@gibiw
gibiw committed Dec 13, 2024
1 parent c2f1d6c commit d1b2bd5
Show file tree
Hide file tree
Showing 6 changed files with 105 additions and 53 deletions.
30 changes: 28 additions & 2 deletions qase-pytest/changelog.md
View file
Original file line number Diff line number Diff line change
@@ -1,3 +1,28 @@
# qase-pytest 6.1.10

## What's new

The ability to override statuses for tests marked with the `xfail` marker has been added. By default, failed tests are
assigned the `skipped` status, and passed tests are assigned the `passed` status. Custom statuses can be specified by
providing the slug of the desired status in the configuration. Configuration values can be set via `qase.config.json` or
environment variables:
- `QASE_PYTEST_XFAIL_STATUS_XFAIL`
- `QASE_PYTEST_XFAIL_STATUS_XPASS`

```diff
{ ...,
"framework": {
"pytest": {
"captureLogs": true,
+ "xfailStatus": {
+ "xfail": "skipped",
+ "xpass": "passed"
+ }
+ }
}
}
```

# qase-pytest 6.1.9

## What's new
Expand All @@ -20,13 +45,14 @@ Fixed an issue with suites [#296]

## What's new

Support new version of qase-python-commons
Support new version of qase-python-commons

# qase-pytest 6.1.6

## What's new

Fixed an issue with the handling video and trace recording for Playwright tests. If a test was part of a class, the video
Fixed an issue with the handling video and trace recording for Playwright tests. If a test was part of a class, the
video
and trace were not attached to the test result.

# qase-pytest 6.1.5
Expand Down
50 changes: 26 additions & 24 deletions qase-pytest/docs/CONFIGURATION.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,34 +11,36 @@ and command line options override both other values.

## Configuration options

| Description | Config file | Environment variable | CLI option | Default value | Required | Possible values |
|------------------------------------------------|--------------------------------|---------------------------------|-----------------------------------|-----------------------------------------|----------|----------------------------|
| Description | Config file | Environment variable | CLI option | Default value | Required | Possible values |
|------------------------------------------------|--------------------------------------|----------------------------------|------------------------------------|-----------------------------------------|----------|----------------------------|
| **Common** |
| Main reporting mode | `mode` | `QASE_MODE` | `--qase-mode` | `testops` | No | `testops`, `report`, `off` |
| Fallback reporting mode | `fallback` | `QASE_FALLBACK` | `--qase-fallback` | `report` | No | `testops`, `report`, `off` |
| Execution plan path | `executionPlan.path` | `QASE_EXECUTION_PLAN_PATH` | `--qase-execution-plan-path` | `./build/qase-execution-plan.json` | No | Any string |
| Qase environment | `environment` | `QASE_ENVIRONMENT` | `--qase-environment` | `local` | No | Any string |
| Root suite | `rootSuite` | `QASE_ROOT_SUITE` | `--qase-root-suite` | | No | Any string |
| Debug logs | `debug` | `QASE_DEBUG` | `--qase-debug` | false | No | `true`, `false` |
| Main reporting mode | `mode` | `QASE_MODE` | `--qase-mode` | `testops` | No | `testops`, `report`, `off` |
| Fallback reporting mode | `fallback` | `QASE_FALLBACK` | `--qase-fallback` | `report` | No | `testops`, `report`, `off` |
| Execution plan path | `executionPlan.path` | `QASE_EXECUTION_PLAN_PATH` | `--qase-execution-plan-path` | `./build/qase-execution-plan.json` | No | Any string |
| Qase environment | `environment` | `QASE_ENVIRONMENT` | `--qase-environment` | `local` | No | Any string |
| Root suite | `rootSuite` | `QASE_ROOT_SUITE` | `--qase-root-suite` | | No | Any string |
| Debug logs | `debug` | `QASE_DEBUG` | `--qase-debug` | false | No | `true`, `false` |
| **Qase TestOps mode configuration** |
| Qase project code | `testops.project` | `QASE_TESTOPS_PROJECT` | `--qase-testops-project` | | Yes | Any string |
| Qase API token | `testops.api.token` | `QASE_TESTOPS_API_TOKEN` | `--qase-testops-api-token` | | Yes | Any string |
| Qase API host | `testops.api.host` | `QASE_TESTOPS_API_HOST` | `--qase-testops-api-host` | `qase.io` | No | Any string |
| Title of the Qase test run | `testops.run.title` | `QASE_TESTOPS_RUN_TITLE` | `--qase-testops-run-title` | `Automated Run {current date and time}` | No | Any string |
| Description of the Qase test run | `testops.run.description` | `QASE_TESTOPS_RUN_DESCRIPTION` | `--qase-testops-run-description` | None, leave empty | No | Any string |
| Create test run using a test plan | `testops.plan.id` | `QASE_TESTOPS_PLAN_ID` | `--qase-testops-plan-id` | None, don't use plans for the test run | No | Any integer |
| Complete test run after running tests | `testops.run.complete` | `QASE_TESTOPS_RUN_COMPLETE` | `--qase-testops-run-complete` | `True` | No | `true`, `false` |
| ID of the Qase test run to report results | `testops.run.id` | `QASE_TESTOPS_RUN_ID` | `--qase-testops-run-id` | None, create a new test run | No | Any integer |
| Batch size for uploading test results | `testops.batch.size` | `QASE_TESTOPS_BATCH_SIZE` | `--qase-testops-batch-size` | 200 | No | 1 to 2000 |
| Create defects in Qase | `testops.defect` | `QASE_TESTOPS_DEFECT` | `--qase-testops-defect` | `False`, don't create defects | No | `True`, `False` |
| Qase project code | `testops.project` | `QASE_TESTOPS_PROJECT` | `--qase-testops-project` | | Yes | Any string |
| Qase API token | `testops.api.token` | `QASE_TESTOPS_API_TOKEN` | `--qase-testops-api-token` | | Yes | Any string |
| Qase API host | `testops.api.host` | `QASE_TESTOPS_API_HOST` | `--qase-testops-api-host` | `qase.io` | No | Any string |
| Title of the Qase test run | `testops.run.title` | `QASE_TESTOPS_RUN_TITLE` | `--qase-testops-run-title` | `Automated Run {current date and time}` | No | Any string |
| Description of the Qase test run | `testops.run.description` | `QASE_TESTOPS_RUN_DESCRIPTION` | `--qase-testops-run-description` | None, leave empty | No | Any string |
| Create test run using a test plan | `testops.plan.id` | `QASE_TESTOPS_PLAN_ID` | `--qase-testops-plan-id` | None, don't use plans for the test run | No | Any integer |
| Complete test run after running tests | `testops.run.complete` | `QASE_TESTOPS_RUN_COMPLETE` | `--qase-testops-run-complete` | `True` | No | `true`, `false` |
| ID of the Qase test run to report results | `testops.run.id` | `QASE_TESTOPS_RUN_ID` | `--qase-testops-run-id` | None, create a new test run | No | Any integer |
| Batch size for uploading test results | `testops.batch.size` | `QASE_TESTOPS_BATCH_SIZE` | `--qase-testops-batch-size` | 200 | No | 1 to 2000 |
| Create defects in Qase | `testops.defect` | `QASE_TESTOPS_DEFECT` | `--qase-testops-defect` | `False`, don't create defects | No | `True`, `False` |
| **Qase Report mode configuration** |
| Local path to store report | `report.connection.path` | `QASE_REPORT_CONNECTION_PATH` | `--qase-report-connection-path` | `./build/qase-report` | No | Any string |
| Report format | `report.connection.format` | `QASE_REPORT_CONNECTION_FORMAT` | `--qase-report-connection-format` | `json` | No | `json`, `jsonp` |
| Driver used for report mode | `report.driver` | `QASE_REPORT_DRIVER` | `--qase-report-driver` | `local` | No | `local` |
| Local path to store report | `report.connection.path` | `QASE_REPORT_CONNECTION_PATH` | `--qase-report-connection-path` | `./build/qase-report` | No | Any string |
| Report format | `report.connection.format` | `QASE_REPORT_CONNECTION_FORMAT` | `--qase-report-connection-format` | `json` | No | `json`, `jsonp` |
| Driver used for report mode | `report.driver` | `QASE_REPORT_DRIVER` | `--qase-report-driver` | `local` | No | `local` |
| **Framework specific options** |
| **Pytest** |
| Capture logs | `framework.pytest.captureLogs` | `QASE_PYTEST_CAPTURE_LOGS` | `--qase-pytest-capture-logs` | `False` | No | `true`, `false` |
| Capture logs | `framework.pytest.captureLogs` | `QASE_PYTEST_CAPTURE_LOGS` | `--qase-pytest-capture-logs` | `False` | No | `true`, `false` |
| XFail status for failed tests | `framework.pytest.xfailStatus.xfail` | `QASE_PYTEST_XFAIL_STATUS_XFAIL` | `--qase-pytest-xfail-status-xfail` | `Skipped` | No | Any string |
| XFail status for passed tests | `framework.pytest.xfailStatus.xpass` | `QASE_PYTEST_XFAIL_STATUS_XPASS` | `--qase-pytest-xfail-status-xpass` | `Passed` | No | Any string |
| **Earlier versions** |
| **qase-pytest v5.x** |
| TestOps bulk (always on since v6) | `testops.bulk` | `QASE_TESTOPS_BULK` | `--qase-testops-bulk` | `True` | No | `true`, `false` |
| Execution chunk size (changed to `batch.size`) | `testops.chunk` | `QASE_TESTOPS_CHUNK` | `--qase-testops-chunk` | 200 | No | 1 to 2000 |
| TestOps bulk (always on since v6) | `testops.bulk` | `QASE_TESTOPS_BULK` | `--qase-testops-bulk` | `True` | No | `true`, `false` |
| Execution chunk size (changed to `batch.size`) | `testops.chunk` | `QASE_TESTOPS_CHUNK` | `--qase-testops-chunk` | 200 | No | 1 to 2000 |
4 changes: 2 additions & 2 deletions qase-pytest/pyproject.toml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"

[project]
name = "qase-pytest"
version = "6.1.9"
version = "6.1.10"
description = "Qase Pytest Plugin for Qase TestOps and Qase Report"
readme = "README.md"
keywords = ["qase", "pytest", "plugin", "testops", "report", "qase reporting", "test observability"]
Expand All @@ -29,7 +29,7 @@ classifiers = [
]
requires-python = ">=3.7"
dependencies = [
"qase-python-commons~=3.2.0",
"qase-python-commons~=3.2.4",
"pytest>=7.4.4",
"filelock~=3.12.2",
"more_itertools",
Expand Down
6 changes: 6 additions & 0 deletions qase-pytest/src/qase/pytest/conftest.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,12 @@ def setup_config_manager(config):
if option == "qase_pytest_capture_logs" and config.option.__dict__[option] is not None:
config_manager.config.framework.pytest.set_capture_logs(config.option.__dict__[option])

if option == "qase_pytest_xfail_status" and config.option.__dict__[option] is not None:
config_manager.config.framework.pytest.xfail_status.set_xfail(config.option.__dict__[option])

if option == "qase_pytest_xpass_status" and config.option.__dict__[option] is not None:
config_manager.config.framework.pytest.xfail_status.set_xpass(config.option.__dict__[option])

return config_manager


Expand Down
16 changes: 16 additions & 0 deletions qase-pytest/src/qase/pytest/options.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -175,6 +175,22 @@ def addoptions(parser, group):
help="Capture logs from pytest"
)

QasePytestOptions.add_option(
parser,
group,
"--qase-pytest-xfail-status-xfail",
dest="qase_pytest_xfail_status",
help="Define xfail status for failed tests. Default: `skipped`"
)

QasePytestOptions.add_option(
parser,
group,
"--qase-pytest-xfail-status-xpass",
dest="qase_pytest_xpass_status",
help="Define xpass status for passed tests. Default: `passed`"
)

@staticmethod
def add_option(parser, group, option, dest, default=None, type=None, **kwargs):
# We are going to add options that were not added before through the manager
Expand Down
52 changes: 27 additions & 25 deletions qase-pytest/src/qase/pytest/plugin.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -141,14 +141,13 @@ def pytest_runtest_logfinish(self):

@pytest.hookimpl(hookwrapper=True)
def pytest_runtest_makereport(self, item, call):
if not self.ignore:
if not self.ignore and call.when == "call":
report = (yield).get_result()

def set_result(res):
self.runtime.result.execution.status = res

def _attach_logs():
# TODO: can be attached twice
if report.caplog:
self.add_attachments((report.caplog, "text/plain", "log.txt"))
if report.capstdout:
Expand All @@ -166,35 +165,32 @@ def _attach_logs():
set_result(PYTEST_TO_QASE_STATUS['FAILED'])
self.runtime.result.add_message(call.excinfo.exconly())
elif report.skipped:
if self.runtime.result.execution.status in (
None,
PYTEST_TO_QASE_STATUS['PASSED'],
):
if self.__is_use_xfail_mark(report):
set_result(self.config.framework.pytest.xfail_status.xfail)
elif self.runtime.result.execution.status in (None, PYTEST_TO_QASE_STATUS['PASSED']):
set_result(PYTEST_TO_QASE_STATUS['SKIPPED'])
else:
if self.runtime.result.execution.status is None:
if self.__is_use_xfail_mark(report):
set_result(self.config.framework.pytest.xfail_status.xpass)
elif self.runtime.result.execution.status is None:
set_result(PYTEST_TO_QASE_STATUS['PASSED'])

if self.reporter.config.framework.pytest.capture_logs and report.when == "call":
if self.reporter.config.framework.pytest.capture_logs:
_attach_logs()

if report.when == "call":
# Attach the video and the trace to the test result
if hasattr(item, 'funcargs') and 'page' in item.funcargs:
page = item.funcargs['page']
if not page.video:
return

folder_name = self.__build_folder_name(item)
output_dir = self.config.framework.playwright.output_dir
base_path = os.path.join(os.getcwd(), output_dir, folder_name)

video_path = os.path.join(base_path, "video.webm")
self.add_attachments(video_path)

if self.config.framework.playwright.trace != Trace.off:
trace_path = os.path.join(base_path, "trace.zip")
self.add_attachments(trace_path)
# Attach the video and the trace to the test result
if hasattr(item, 'funcargs') and 'page' in item.funcargs:
page = item.funcargs['page']
if not page.video:
return
folder_name = self.__build_folder_name(item)
output_dir = self.config.framework.playwright.output_dir
base_path = os.path.join(os.getcwd(), output_dir, folder_name)
video_path = os.path.join(base_path, "video.webm")
self.add_attachments(video_path)
if self.config.framework.playwright.trace != Trace.off:
trace_path = os.path.join(base_path, "trace.zip")
self.add_attachments(trace_path)
else:
yield

Expand Down Expand Up @@ -391,6 +387,12 @@ def __build_folder_name(item):
def __sanitize_path_component(component):
return component.replace(os.sep, "-").replace(".", "-").replace("_", "-").lower()

@staticmethod
def __is_use_xfail_mark(report):
if report.keywords.get("xfail"):
return True
return False


class QasePytestPluginSingleton:
_instance = None
Expand Down

0 comments on commit d1b2bd5

Please sign in to comment.