From da4c0152b531f3d4abd4489558488a5864994ea0 Mon Sep 17 00:00:00 2001 From: olensmar Date: Mon, 9 Jan 2023 14:36:19 +0100 Subject: [PATCH 01/37] Initial WIP version of Testing Events --- testing-events.md | 141 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 141 insertions(+) create mode 100644 testing-events.md diff --git a/testing-events.md b/testing-events.md new file mode 100644 index 00000000..bb6f76e6 --- /dev/null +++ b/testing-events.md @@ -0,0 +1,141 @@ + +# Testing Events + +Testing events include the subject and predicates related to test-execution and reporting activities which are +performed out-of-bounds of CI/CD pipelines/workflows. For test-related events performed as part of a +CI pipeline have a look at [Continuous Integration Events](continuous-integration-pipeline-events.md) + +## Subjects + +This specification defines three subjects in this stage: `testCase`, `testSuite` and `testArtifact`. + +| Subject | Description | Predicates | +|---------------------------------|--------------------------------|------------------------------------------------------------------------------------------------| +| [`testCase`](#testcase) | A software test case | [`queued`](#testcase-queued), [`started`](#testcase-started), [`finished`](#testcase-finished) | +| [`testSuite`](#testsuite) | A collection of test cases | [`started`](#testsuite-started), [`finished`](#testsuite-finished) | +| [`testArtifact`](#testartifact) | An artifact produced by a test | [`published`](#testartifact-published) | + +### `testCase` + +A `testCase` is a process that performs a test against an input software artifact of some kind, for instance source code, a binary, a container image or else. A `testCase` is the smallest unit of testing that the user wants to track. `testCases` are executed, and `testSuites` are for grouping purposes. For this reason, `testCases` can be queued. + +| Field | Type | Description | Examples | +|-----------|-----------------|-------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------| +| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | `staging/tekton`, `tekton-dev-123` | +| type | `String` | Type type of test, must be one of the following: `unit`, `functional`, `performance`, `integration`, `security`, `compliance`, `other` | `functional` | +| testSuite | `String` | an optional ID of the containing testSuite | `functional` | + +### `testSuite` + +A `testSuite` represents a set of one or more `testCases`. + +__Note:__ The data model for `testSuite` only includes `id` and `source`, inputs and outputs of the process are not specified yet, as well as the relation to `testCase`. + +| Field | Type | Description | Examples | +|-------|------|-------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `unit`, `e2e`, `security` | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | `staging/tekton`, `tekton-dev-123`| + +### `testArtifact` + +An `testArtifact` is usually produced as output of a test execution + +| Field | Type | Description | Examples | +|--------|-----------------|---------------------------------------------------------------|---------------------------------------------------------| +| id | `String` | Uniquely identifies the subject within the source. | `23123123` | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | `staging/tekton`, `tekton-dev-123` | +| type | `String` | The type of artifact, one of report, video, image, log, other | `video` | +| format | `String` | The Content-Type of the artifact | `application/pdf`, `image/png`, `application/json` | +| url | `URI-Reference` | A reference to the specified artifact | `https://testkube.mycluster.internal/artifacts/23123123` | + +## Events + +### `testCase queued` + +This event represents a Test task that has been queued, and it is waiting to be started. + +- Event Type: __`dev.cdevents.testcase.queued.0.1.0`__ +- Predicate: queued +- Subject: [`testCase`](#testcase) + +| Field | Type | Description | Examples | Required | +|-------|------|-------------|----------|----------------------------| +| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | + +### `testCase started` + +This event represents a Test task that has started. + +- Event Type: __`dev.cdevents.testcase.started.0.1.0`__ +- Predicate: started +- Subject: [`testCase`](#testcase) + +| Field | Type | Description | Examples | Required | +|-------|------|-------------|----------|----------------------------| +| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | + +### `testCase finished` + +This event represents a Test task that has finished. This event will eventually contain the finished status: success, error or failure. + +- Event Type: __`dev.cdevents.testcase.finished.0.1.0`__ +- Predicate: finished +- Subject: [`testCase`](#testcase) + +| Field | Type | Description | Examples | Required | +|-------|------|-------------|----------|---------------------| +| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| status | `String` | The status of the testSuite execution, one of `passed`, `failed`, `aborted` | | ✅ | + +### `testSuite started` + +This event represents a Test suite that has been started. + +- Event Type: __`dev.cdevents.testsuite.started.0.1.0`__ +- Predicate: started +- Subject: [`testSuite`](#testsuite) + +| Field | Type | Description | Examples | Required | +|-------|------|-------------|----------|----------------------------| +| id | `String` | Uniquely identifies the subject within the source. | `unit`, `e2e`, `security` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | + +### `testSuite finished` + +This event represents a Test suite that has has finished, the event will contain the finished status: success, error or failure. + +- Event Type: __`dev.cdevents.testsuite.finished.0.1.0`__ +- Predicate: finished +- Subject: [`testSuite`](#testsuite) + +| Field | Type | Description | Examples | Required | +|-------|------|-------------|----------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `unit`, `e2e`, `security` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| status | `String` | The status of the testSuite execution, one of `passed`, `failed`, `aborted` | | ✅ | + +### `testArtifact published` + +The event represents a test artifact that has been published and can be consumed by others. + +- Event Type: __`dev.cdevents.testartifact.published.0.1.0`__ +- Predicate: published +- Subject: [`testArtifact`](#testartifact) + +| Field | Type | Description | Examples | Required | +|-------|------|-------------|----------|----------------------------| +| id | `Purl` | Uniquely identifies the subject within the source. | `pkg:oci/myapp@sha256%3A0b31b1c02ff458ad9b7b81cbdf8f028bd54699fa151f221d1e8de6817db93427?repository_url=mycr.io/myapp`, `pkg:golang/mygit.com/myorg/myapp@234fd47e07d1004f0aed9c` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | From fe695c160044bb34e97215f510b913a11c328d3e Mon Sep 17 00:00:00 2001 From: olensmar Date: Mon, 9 Jan 2023 17:29:03 +0100 Subject: [PATCH 02/37] fixes, removed queued event, added executionId --- testing-events.md | 123 +++++++++++++++++++++------------------------- 1 file changed, 57 insertions(+), 66 deletions(-) diff --git a/testing-events.md b/testing-events.md index bb6f76e6..740731a4 100644 --- a/testing-events.md +++ b/testing-events.md @@ -1,78 +1,64 @@ # Testing Events -Testing events include the subject and predicates related to test-execution and reporting activities which are -performed out-of-bounds of CI/CD pipelines/workflows. For test-related events performed as part of a -CI pipeline have a look at [Continuous Integration Events](continuous-integration-pipeline-events.md) +Testing events covers the subjects and predicates related to test-execution +which is performed independently of CI/CD pipelines. ## Subjects This specification defines three subjects in this stage: `testCase`, `testSuite` and `testArtifact`. -| Subject | Description | Predicates | -|---------------------------------|--------------------------------|------------------------------------------------------------------------------------------------| -| [`testCase`](#testcase) | A software test case | [`queued`](#testcase-queued), [`started`](#testcase-started), [`finished`](#testcase-finished) | -| [`testSuite`](#testsuite) | A collection of test cases | [`started`](#testsuite-started), [`finished`](#testsuite-finished) | -| [`testArtifact`](#testartifact) | An artifact produced by a test | [`published`](#testartifact-published) | +| Subject | Description | Predicates | +|---------------------------------|--------------------------------|--------------------------------------------------------------------| +| [`testCase`](#testcase) | A software test case | [`started`](#testcase-started), [`finished`](#testcase-finished) | +| [`testSuite`](#testsuite) | A collection of test cases | [`started`](#testsuite-started), [`finished`](#testsuite-finished) | +| [`testArtifact`](#testartifact) | An artifact produced by a test | [`published`](#testartifact-published) | ### `testCase` -A `testCase` is a process that performs a test against an input software artifact of some kind, for instance source code, a binary, a container image or else. A `testCase` is the smallest unit of testing that the user wants to track. `testCases` are executed, and `testSuites` are for grouping purposes. For this reason, `testCases` can be queued. +A `testCase` is a process that performs a test against an input software artifact of some kind, for instance source code, a binary, a container image or else. +A `testCase` is the smallest unit of testing that the user wants to track. `testCases` are executed, and `testSuites` are for grouping purposes. -| Field | Type | Description | Examples | -|-----------|-----------------|-------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------| -| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | `staging/tekton`, `tekton-dev-123` | -| type | `String` | Type type of test, must be one of the following: `unit`, `functional`, `performance`, `integration`, `security`, `compliance`, `other` | `functional` | -| testSuite | `String` | an optional ID of the containing testSuite | `functional` | +| Field | Type | Description | Examples | +|-----------|-----------------|----------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------| +| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | `staging/tekton`, `tekton-dev-123` | +| type | `String` | Type type of test, must be one of the following: `unit`, `functional`, `performance`, `integration`, `security`, `compliance`, `other` | `functional` | +| url | `URI-Reference` | An optional reference to view/access the specified testCase | `https://testkube.mycluster.internal/testCases/untitest-abc` | ### `testSuite` A `testSuite` represents a set of one or more `testCases`. -__Note:__ The data model for `testSuite` only includes `id` and `source`, inputs and outputs of the process are not specified yet, as well as the relation to `testCase`. - -| Field | Type | Description | Examples | -|-------|------|-------------|----------| -| id | `String` | Uniquely identifies the subject within the source. | `unit`, `e2e`, `security` | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | `staging/tekton`, `tekton-dev-123`| +| Field | Type | Description | Examples | +|--------|-----------------|--------------------------------------------------------------|---------------------------------------------------------------| +| id | `String` | Uniquely identifies the subject within the source. | `my-testsuite`, `regression-123` | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | `staging/tekton`, `tekton-dev-123` | +| url | `URI-Reference` | An optional reference to view/access the specified testSuite | `https://testkube.mycluster.internal/testSuites/my-testsuite` | ### `testArtifact` -An `testArtifact` is usually produced as output of a test execution +One ore more `testArtifact`s are usually produced as the result of a test execution. -| Field | Type | Description | Examples | -|--------|-----------------|---------------------------------------------------------------|---------------------------------------------------------| -| id | `String` | Uniquely identifies the subject within the source. | `23123123` | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | `staging/tekton`, `tekton-dev-123` | -| type | `String` | The type of artifact, one of report, video, image, log, other | `video` | -| format | `String` | The Content-Type of the artifact | `application/pdf`, `image/png`, `application/json` | -| url | `URI-Reference` | A reference to the specified artifact | `https://testkube.mycluster.internal/artifacts/23123123` | +| Field | Type | Description | Examples | +|--------|-----------------|-------------------------------------------------------------------------|----------------------------------------------------------| +| id | `String` | Uniquely identifies the subject within the source. | `23123123` | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | `staging/tekton`, `tekton-dev-123` | +| type | `String` | The type of artifact, one of `report`, `video`, `image`, `log`, `other` | `video` | +| format | `String` | The Content-Type of the artifact | `application/pdf`, `image/png`, `application/json` | +| url | `URI-Reference` | A reference to retrieve the specified artifact | `https://testkube.mycluster.internal/artifacts/23123123` | ## Events -### `testCase queued` - -This event represents a Test task that has been queued, and it is waiting to be started. - -- Event Type: __`dev.cdevents.testcase.queued.0.1.0`__ -- Predicate: queued -- Subject: [`testCase`](#testcase) - -| Field | Type | Description | Examples | Required | -|-------|------|-------------|----------|----------------------------| -| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | - ### `testCase started` This event represents a Test task that has started. @@ -81,10 +67,12 @@ This event represents a Test task that has started. - Predicate: started - Subject: [`testCase`](#testcase) -| Field | Type | Description | Examples | Required | -|-------|------|-------------|----------|----------------------------| -| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| Field | Type | Description | Examples | Required | +|-------------|-----------------|--------------------------------------------------------------------------------------------|-------------------------------------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| executionId | `String` | An optional execution ID to enable handling of multiple simultaneous executions | | | +| testSuiteId | `String` | An optional `testSuite` ID if this `testCase` was started as part of a testSuite execution | | | ### `testCase finished` @@ -94,11 +82,12 @@ This event represents a Test task that has finished. This event will eventually - Predicate: finished - Subject: [`testCase`](#testcase) -| Field | Type | Description | Examples | Required | -|-------|------|-------------|----------|---------------------| -| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | -| status | `String` | The status of the testSuite execution, one of `passed`, `failed`, `aborted` | | ✅ | +| Field | Type | Description | Examples | Required | +|-------------|-----------------|---------------------------------------------------------------------------------|-------------------------------------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| executionId | `String` | An optional execution ID to enable handling of multiple simultaneous executions | | | +| status | `String` | The status of the testSuite execution, one of `passed`, `failed`, `aborted` | | ✅ | ### `testSuite started` @@ -108,10 +97,11 @@ This event represents a Test suite that has been started. - Predicate: started - Subject: [`testSuite`](#testsuite) -| Field | Type | Description | Examples | Required | -|-------|------|-------------|----------|----------------------------| -| id | `String` | Uniquely identifies the subject within the source. | `unit`, `e2e`, `security` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| Field | Type | Description | Examples | Required | +|-------------|-----------------|---------------------------------------------------------------------------------|---------------------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `unit`, `e2e`, `security` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| executionId | `String` | An optional execution ID to enable handling of multiple simultaneous executions | | | ### `testSuite finished` @@ -121,11 +111,11 @@ This event represents a Test suite that has has finished, the event will contain - Predicate: finished - Subject: [`testSuite`](#testsuite) -| Field | Type | Description | Examples | Required | -|-------|------|-------------|----------|----------| -| id | `String` | Uniquely identifies the subject within the source. | `unit`, `e2e`, `security` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | -| status | `String` | The status of the testSuite execution, one of `passed`, `failed`, `aborted` | | ✅ | +| Field | Type | Description | Examples | Required | +|--------|-----------------|-----------------------------------------------------------------------------|---------------------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `unit`, `e2e`, `security` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| status | `String` | The status of the testSuite execution, one of `passed`, `failed`, `aborted` | | ✅ | ### `testArtifact published` @@ -135,7 +125,8 @@ The event represents a test artifact that has been published and can be consumed - Predicate: published - Subject: [`testArtifact`](#testartifact) -| Field | Type | Description | Examples | Required | -|-------|------|-------------|----------|----------------------------| -| id | `Purl` | Uniquely identifies the subject within the source. | `pkg:oci/myapp@sha256%3A0b31b1c02ff458ad9b7b81cbdf8f028bd54699fa151f221d1e8de6817db93427?repository_url=mycr.io/myapp`, `pkg:golang/mygit.com/myorg/myapp@234fd47e07d1004f0aed9c` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| Field | Type | Description | Examples | Required | +|-------------|-----------------|--------------------------------------------------------------------------------------------------|------------|----------| +| id | `Purl` | Uniquely identifies the subject within the source. | `12312334` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| executionId | `String` | An optional execution ID to link this artifact to a specific `testCase` or `testSuite` execution | | | From 965543f8dacf421123c97fcb1c458d00488bf8db Mon Sep 17 00:00:00 2001 From: olensmar Date: Tue, 10 Jan 2023 08:30:38 +0100 Subject: [PATCH 03/37] added optional reason to finished events, loosened testCase type field --- testing-events.md | 25 ++++++++++++++----------- 1 file changed, 14 insertions(+), 11 deletions(-) diff --git a/testing-events.md b/testing-events.md index 740731a4..cdd2e022 100644 --- a/testing-events.md +++ b/testing-events.md @@ -28,12 +28,12 @@ This specification defines three subjects in this stage: `testCase`, `testSuite` A `testCase` is a process that performs a test against an input software artifact of some kind, for instance source code, a binary, a container image or else. A `testCase` is the smallest unit of testing that the user wants to track. `testCases` are executed, and `testSuites` are for grouping purposes. -| Field | Type | Description | Examples | -|-----------|-----------------|----------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------| -| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | `staging/tekton`, `tekton-dev-123` | -| type | `String` | Type type of test, must be one of the following: `unit`, `functional`, `performance`, `integration`, `security`, `compliance`, `other` | `functional` | -| url | `URI-Reference` | An optional reference to view/access the specified testCase | `https://testkube.mycluster.internal/testCases/untitest-abc` | +| Field | Type | Description | Examples | +|--------|-----------------|-------------------------------------------------------------|--------------------------------------------------------------| +| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | `staging/tekton`, `tekton-dev-123` | +| type | `String` | An optional type of test | `functional`, `unit`, `performance`, `security` | +| url | `URI-Reference` | An optional reference to view/access the specified testCase | `https://testkube.mycluster.internal/testCases/untitest-abc` | ### `testSuite` @@ -88,6 +88,8 @@ This event represents a Test task that has finished. This event will eventually | source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | | executionId | `String` | An optional execution ID to enable handling of multiple simultaneous executions | | | | status | `String` | The status of the testSuite execution, one of `passed`, `failed`, `aborted` | | ✅ | +| reason | `String` | An optional reason related to the status of the execution | `Cancelled by user`, `Failed assertion` | | + ### `testSuite started` @@ -111,11 +113,12 @@ This event represents a Test suite that has has finished, the event will contain - Predicate: finished - Subject: [`testSuite`](#testsuite) -| Field | Type | Description | Examples | Required | -|--------|-----------------|-----------------------------------------------------------------------------|---------------------------|----------| -| id | `String` | Uniquely identifies the subject within the source. | `unit`, `e2e`, `security` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | -| status | `String` | The status of the testSuite execution, one of `passed`, `failed`, `aborted` | | ✅ | +| Field | Type | Description | Examples | Required | +|--------|-----------------|-----------------------------------------------------------------------------|----------------------------------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `unit`, `e2e`, `security` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| status | `String` | The status of the testSuite execution, one of `passed`, `failed`, `aborted` | | ✅ | +| reason | `String` | An optional reason related to the status of the execution | `Cancelled by user`, `Failed testCase` | | ### `testArtifact published` From 8fc7ef247164096f15931806252815bdfef048a2 Mon Sep 17 00:00:00 2001 From: olensmar Date: Tue, 10 Jan 2023 09:57:42 +0100 Subject: [PATCH 04/37] added severity to testCase --- testing-events.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/testing-events.md b/testing-events.md index cdd2e022..df6d4e68 100644 --- a/testing-events.md +++ b/testing-events.md @@ -28,12 +28,13 @@ This specification defines three subjects in this stage: `testCase`, `testSuite` A `testCase` is a process that performs a test against an input software artifact of some kind, for instance source code, a binary, a container image or else. A `testCase` is the smallest unit of testing that the user wants to track. `testCases` are executed, and `testSuites` are for grouping purposes. -| Field | Type | Description | Examples | -|--------|-----------------|-------------------------------------------------------------|--------------------------------------------------------------| -| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | `staging/tekton`, `tekton-dev-123` | -| type | `String` | An optional type of test | `functional`, `unit`, `performance`, `security` | -| url | `URI-Reference` | An optional reference to view/access the specified testCase | `https://testkube.mycluster.internal/testCases/untitest-abc` | +| Field | Type | Description | Examples | +|----------|-----------------|------------------------------------------------------------------------|--------------------------------------------------------------| +| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | `staging/tekton`, `tekton-dev-123` | +| type | `String` | An optional type of test | `functional`, `unit`, `performance`, `security` | +| severity | `String` | An optional severity, one of `warn`, `critical`, `low`, `high`, `info` | `functional`, `unit`, `performance`, `security` | +| url | `URI-Reference` | An optional reference to view/access the specified testCase | `https://testkube.mycluster.internal/testCases/untitest-abc` | ### `testSuite` From 31e52fe8e2c575a1f8d3aa37092c247edf9396f0 Mon Sep 17 00:00:00 2001 From: Bruno Lopes Date: Fri, 13 Jan 2023 16:31:10 +0400 Subject: [PATCH 05/37] Update testing-events.md --- testing-events.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/testing-events.md b/testing-events.md index df6d4e68..00ca3969 100644 --- a/testing-events.md +++ b/testing-events.md @@ -30,11 +30,11 @@ A `testCase` is the smallest unit of testing that the user wants to track. `test | Field | Type | Description | Examples | |----------|-----------------|------------------------------------------------------------------------|--------------------------------------------------------------| -| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | +| id | `String` | Uniquely identifies the subject within the source. | `integration-test-abc`, `e2e-test1`, `scan-image1` | | source | `URI-Reference` | [source](spec.md#source--context-) from the context | `staging/tekton`, `tekton-dev-123` | | type | `String` | An optional type of test | `functional`, `unit`, `performance`, `security` | -| severity | `String` | An optional severity, one of `warn`, `critical`, `low`, `high`, `info` | `functional`, `unit`, `performance`, `security` | -| url | `URI-Reference` | An optional reference to view/access the specified testCase | `https://testkube.mycluster.internal/testCases/untitest-abc` | +| severity | `String` | An optional severity, one of `critical`, `low`, `medium`, `high` | `critical`, `low`, `medium`, `high` | +| url | `URI-Reference` | An optional reference to view/access the specified testCase | `https://testkube.mycluster.internal/testCases/untitest-abc` | ### `testSuite` @@ -58,6 +58,10 @@ One ore more `testArtifact`s are usually produced as the result of a test execut | format | `String` | The Content-Type of the artifact | `application/pdf`, `image/png`, `application/json` | | url | `URI-Reference` | A reference to retrieve the specified artifact | `https://testkube.mycluster.internal/artifacts/23123123` | +| testCaseId | `URI-Reference` | A reference to the testCase +`https://testkube.mycluster.internal/artifacts/23123123` | + + ## Events ### `testCase started` From 2a728227ac30aabff0eb1edfcaa1f52110c1a67f Mon Sep 17 00:00:00 2001 From: olensmar Date: Sun, 15 Jan 2023 21:23:08 +0100 Subject: [PATCH 06/37] formatting and added Test Events to README.md --- README.md | 4 ++++ testing-events.md | 23 ++++++++++------------- 2 files changed, 14 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index 0e044521..30177f5b 100644 --- a/README.md +++ b/README.md @@ -82,6 +82,10 @@ Handling Events associated with Continuous Deployment activities Handling Events associated with the health of the services deployed and running in a specific environment +### [Testing Events](./testing-events.md) + +Handling Events associated with Test execution performed independently of CI/CD pipelines. + ### [CloudEvents Binding and Transport](./cloudevents-binding.md) Defining how CDEvents are mapped to CloudEvents for transportation and delivery diff --git a/testing-events.md b/testing-events.md index 00ca3969..fcc25e73 100644 --- a/testing-events.md +++ b/testing-events.md @@ -11,7 +11,7 @@ description: > # Testing Events Testing events covers the subjects and predicates related to test-execution -which is performed independently of CI/CD pipelines. +performed independently of CI/CD pipelines. ## Subjects @@ -48,19 +48,16 @@ A `testSuite` represents a set of one or more `testCases`. ### `testArtifact` -One ore more `testArtifact`s are usually produced as the result of a test execution. - -| Field | Type | Description | Examples | -|--------|-----------------|-------------------------------------------------------------------------|----------------------------------------------------------| -| id | `String` | Uniquely identifies the subject within the source. | `23123123` | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | `staging/tekton`, `tekton-dev-123` | -| type | `String` | The type of artifact, one of `report`, `video`, `image`, `log`, `other` | `video` | -| format | `String` | The Content-Type of the artifact | `application/pdf`, `image/png`, `application/json` | -| url | `URI-Reference` | A reference to retrieve the specified artifact | `https://testkube.mycluster.internal/artifacts/23123123` | - -| testCaseId | `URI-Reference` | A reference to the testCase -`https://testkube.mycluster.internal/artifacts/23123123` | +One or more `testArtifact`s are usually produced as the result of a test execution. +| Field | Type | Description | Examples | +|------------|-----------------|-------------------------------------------------------------------------|----------------------------------------------------------| +| id | `String` | Uniquely identifies the subject within the source. | `23123123` | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | `staging/tekton`, `tekton-dev-123` | +| type | `String` | The type of artifact, one of `report`, `video`, `image`, `log`, `other` | `video` | +| format | `String` | The Content-Type of the artifact | `application/pdf`, `image/png`, `application/json` | +| url | `URI-Reference` | A reference to retrieve the specified artifact | `https://testkube.mycluster.internal/artifacts/23123123` | +| testCaseId | `URI-Reference` | An optional reference to the testCase resulting in this testArtifact | `https://testkube.mycluster.internal/testCase/123123` | ## Events From 2efb0d4825d8f45bfe687934bac86645e84a1f5f Mon Sep 17 00:00:00 2001 From: olensmar Date: Tue, 17 Jan 2023 11:22:16 +0100 Subject: [PATCH 07/37] - testing-events updates --- continuous-integration.md | 103 ++------------------------------------ testing-events.md | 85 +++++++++++++++---------------- 2 files changed, 47 insertions(+), 141 deletions(-) diff --git a/continuous-integration.md b/continuous-integration.md index 3766703d..d030a125 100644 --- a/continuous-integration.md +++ b/continuous-integration.md @@ -19,10 +19,10 @@ This specification defines three subjects in this stage: `builds`, `artifacts` a | Subject | Description | Predicates | |---------|-------------|------------| | [`build`](#build) | A software build | [`queued`](#build-queued), [`started`](#build-started), [`finished`](#build-finished)| -| [`testCase`](#testcase) | A software test case | [`queued`](#testcase-queued), [`started`](#testcase-started), [`finished`](#testcase-finished)| -| [`testSuite`](#testsuite) | A collection of test cases | [`started`](#testsuite-started), [`finished`](#testsuite-finished)| | [`artifact`](#artifact) | An artifact produced by a build | [`packaged`](#artifact-packaged), [`published`](#artifact-published)| +> `testCase`/`testSuite` events have moved to their own top-level bucket [Testing Events](testing-events.md) + ### `build` A `build` is a process that uses a recipe to produce an artifact from source code. @@ -31,35 +31,10 @@ __Note:__ The data model for `builds`, apart from `id` and `source`, only includ | Field | Type | Description | Examples | |-------|------|-------------|----------| -| id | `String` | See [id](spec.md#id-subject)| `1234`, `maven123`, `builds/taskrun123` | -| source | `URI-Reference` | See [source](spec.md#source-subject) | `staging/tekton`, `tekton-dev-123`| -| type | `String` | See [type](spec.md#type-subject) | `build` | +| id | `String` | Uniquely identifies the subject within the source. | `1234`, `maven123`, `builds/taskrun123` | +| source | `URI-Reference` | [source](../spec.md#source) from the context | `staging/tekton`, `tekton-dev-123`| | artifactId | `String` | Identifier of the artifact produced by the build | `pkg:oci/myapp@sha256%3A0b31b1c02ff458ad9b7b81cbdf8f028bd54699fa151f221d1e8de6817db93427`, `pkg:golang/mygit.com/myorg/myapp@234fd47e07d1004f0aed9c` | -### `testCase` - -A `testCase` is a process that performs a test against an input software artifact of some kind, for instance source code, a binary, a container image or else. A `testCase` is the smallest unit of testing that the user wants to track. `testCases` are executed, and `testSuites` are for grouping purposes. For this reason, `testCases` can be queued. - -__Note:__ The data model for `testCase` only includes `id` and `source`, inputs and outputs of the process are not specified yet, as well as the relation to `testSuite`. - -| Field | Type | Description | Examples | -|-------|------|-------------|----------| -| id | `String` | See [id](spec.md#id-subject)| `unitest-abc`, `e2e-test1`, `scan-image1` | -| source | `URI-Reference` | See [source](spec.md#source-subject) | `staging/tekton`, `tekton-dev-123`| -| type | `String` | See [type](spec.md#type-subject) | `testCase` | - -### `testSuite` - -A `testSuite` represents a set of one or more `testCases`. - -__Note:__ The data model for `testSuite` only includes `id` and `source`, inputs and outputs of the process are not specified yet, as well as the relation to `testCase`. - -| Field | Type | Description | Examples | -|-------|------|-------------|----------| -| id | `String` | See [id](spec.md#id-subject)| `unit`, `e2e`, `security` | -| source | `URI-Reference` | See [source](spec.md#source-subject) | `staging/tekton`, `tekton-dev-123`| -| type | `String` | See [type](spec.md#type-subject) | `testSuite` | - ### `artifact` An `artifact` is usually produced as output of a build process. Events need to be generated to indicate that an `artifact` has been packaged and released for others to use. These events can be produced by the artifact producer or by the artifact storage system. @@ -116,76 +91,6 @@ This event represents a Build task that has finished. This event will eventually | type | `String` | See [type](spec.md#type-subject) | | | artifactId | `Purl` | Identifier of the artifact produced by the build | `pkg:oci/myapp@sha256%3A0b31b1c02ff458ad9b7b81cbdf8f028bd54699fa151f221d1e8de6817db93427`, `pkg:golang/mygit.com/myorg/myapp@234fd47e07d1004f0aed9c` | `build` | | -### `testCase queued` - -This event represents a Test task that has been queued, and it is waiting to be started. - -- Event Type: __`dev.cdevents.testcase.queued.0.1.1`__ -- Predicate: queued -- Subject: [`testCase`](#testcase) - -| Field | Type | Description | Examples | Required | -|-------|------|-------------|----------|----------------------------| -| id | `String` | See [id](spec.md#id-subject)| `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | -| source | `URI-Reference` | See [source](spec.md#source-subject) | | | -| type | `String` | See [type](spec.md#type-subject) | `testCase` | | - -### `testCase started` - -This event represents a Test task that has started. - -- Event Type: __`dev.cdevents.testcase.started.0.1.1`__ -- Predicate: started -- Subject: [`testCase`](#testcase) - -| Field | Type | Description | Examples | Required | -|-------|------|-------------|----------|----------------------------| -| id | `String` | See [id](spec.md#id-subject)| `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | -| source | `URI-Reference` | See [source](spec.md#source-subject) | | | -| type | `String` | See [type](spec.md#type-subject) | `testCase` | | - -### `testCase finished` - -This event represents a Test task that has finished. This event will eventually contain the finished status: success, error or failure. - -- Event Type: __`dev.cdevents.testcase.finished.0.1.1`__ -- Predicate: finished -- Subject: [`testCase`](#testcase) - -| Field | Type | Description | Examples | Required | -|-------|------|-------------|----------|----------------------------| -| id | `String` | See [id](spec.md#id-subject)| `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | -| source | `URI-Reference` | See [source](spec.md#source-subject) | | | -| type | `String` | See [type](spec.md#type-subject) | `testCase` | | - -### `testSuite started` - -This event represents a Test suite that has been started. - -- Event Type: __`dev.cdevents.testsuite.started.0.1.1`__ -- Predicate: started -- Subject: [`testSuite`](#testsuite) - -| Field | Type | Description | Examples | Required | -|-------|------|-------------|----------|----------------------------| -| id | `String` | See [id](spec.md#id-subject)| `unit`, `e2e`, `security` | ✅ | -| source | `URI-Reference` | See [source](spec.md#source-subject) | | | -| type | `String` | See [type](spec.md#type-subject) | `testSuite` | | - -### `testSuite finished` - -This event represents a Test suite that has has finished, the event will contain the finished status: success, error or failure. - -- Event Type: __`dev.cdevents.testsuite.finished.0.1.1`__ -- Predicate: finished -- Subject: [`testSuite`](#testsuite) - -| Field | Type | Description | Examples | Required | -|-------|------|-------------|----------|----------------------------| -| id | `String` | See [id](spec.md#id-subject)| `unit`, `e2e`, `security` | ✅ | -| source | `URI-Reference` | See [source](spec.md#source-subject) | | | -| type | `String` | See [type](spec.md#type-subject) | `testSuite` | | - ### `artifact packaged` The event represents an artifact that has been packaged for distribution; this artifact is now versioned with a fixed version. diff --git a/testing-events.md b/testing-events.md index fcc25e73..603c96bb 100644 --- a/testing-events.md +++ b/testing-events.md @@ -17,24 +17,23 @@ performed independently of CI/CD pipelines. This specification defines three subjects in this stage: `testCase`, `testSuite` and `testArtifact`. -| Subject | Description | Predicates | -|---------------------------------|--------------------------------|--------------------------------------------------------------------| -| [`testCase`](#testcase) | A software test case | [`started`](#testcase-started), [`finished`](#testcase-finished) | -| [`testSuite`](#testsuite) | A collection of test cases | [`started`](#testsuite-started), [`finished`](#testsuite-finished) | -| [`testArtifact`](#testartifact) | An artifact produced by a test | [`published`](#testartifact-published) | +| Subject | Description | Predicates | +|-----------------------------|-------------------------------------------------|--------------------------------------------------------------------| +| [`testCase`](#testcase) | A software test case | [`started`](#testcase-started), [`finished`](#testcase-finished) | +| [`testSuite`](#testsuite) | A collection of test cases | [`started`](#testsuite-started), [`finished`](#testsuite-finished) | +| [`testOutput`](#testoutput) | An output artifact produced by a test execution | [`published`](#testoutput-published) | ### `testCase` A `testCase` is a process that performs a test against an input software artifact of some kind, for instance source code, a binary, a container image or else. A `testCase` is the smallest unit of testing that the user wants to track. `testCases` are executed, and `testSuites` are for grouping purposes. -| Field | Type | Description | Examples | -|----------|-----------------|------------------------------------------------------------------------|--------------------------------------------------------------| -| id | `String` | Uniquely identifies the subject within the source. | `integration-test-abc`, `e2e-test1`, `scan-image1` | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | `staging/tekton`, `tekton-dev-123` | -| type | `String` | An optional type of test | `functional`, `unit`, `performance`, `security` | -| severity | `String` | An optional severity, one of `critical`, `low`, `medium`, `high` | `critical`, `low`, `medium`, `high` | -| url | `URI-Reference` | An optional reference to view/access the specified testCase | `https://testkube.mycluster.internal/testCases/untitest-abc` | +| Field | Type | Description | Examples | +|--------|-----------------|-------------------------------------------------------------|--------------------------------------------------------------| +| id | `String` | Uniquely identifies the subject within the source. | `integration-test-abc`, `e2e-test1`, `scan-image1` | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | `staging/tekton`, `tekton-dev-123` | +| type | `String` | An optional type of test | `functional`, `unit`, `performance`, `security` | +| url | `URI-Reference` | An optional reference to view/access the specified testCase | `https://testkube.mycluster.internal/testCases/untitest-abc` | ### `testSuite` @@ -46,26 +45,26 @@ A `testSuite` represents a set of one or more `testCases`. | source | `URI-Reference` | [source](spec.md#source--context-) from the context | `staging/tekton`, `tekton-dev-123` | | url | `URI-Reference` | An optional reference to view/access the specified testSuite | `https://testkube.mycluster.internal/testSuites/my-testsuite` | -### `testArtifact` +### `testOutput` -One or more `testArtifact`s are usually produced as the result of a test execution. +One or more `testOutput` artifacts are usually produced as the result of a test execution. | Field | Type | Description | Examples | |------------|-----------------|-------------------------------------------------------------------------|----------------------------------------------------------| | id | `String` | Uniquely identifies the subject within the source. | `23123123` | | source | `URI-Reference` | [source](spec.md#source--context-) from the context | `staging/tekton`, `tekton-dev-123` | -| type | `String` | The type of artifact, one of `report`, `video`, `image`, `log`, `other` | `video` | -| format | `String` | The Content-Type of the artifact | `application/pdf`, `image/png`, `application/json` | -| url | `URI-Reference` | A reference to retrieve the specified artifact | `https://testkube.mycluster.internal/artifacts/23123123` | -| testCaseId | `URI-Reference` | An optional reference to the testCase resulting in this testArtifact | `https://testkube.mycluster.internal/testCase/123123` | +| type | `String` | The type of output, one of `report`, `video`, `image`, `log`, `other` | `video` | +| format | `String` | The Content-Type of the output artifact | `application/pdf`, `image/png`, `application/json` | +| url | `URI-Reference` | A reference to retrieve the specified output artifact | `https://testkube.mycluster.internal/artifacts/23123123` | +| testCaseId | `URI-Reference` | An optional reference to the testCase resulting in this output artifact | `https://testkube.mycluster.internal/testCase/123123` | ## Events ### `testCase started` -This event represents a Test task that has started. +This event represents a started testCase execution. -- Event Type: __`dev.cdevents.testcase.started.0.1.0`__ +- Event Type: __`dev.cdevents.testcase.started.0.1.1`__ - Predicate: started - Subject: [`testCase`](#testcase) @@ -78,24 +77,25 @@ This event represents a Test task that has started. ### `testCase finished` -This event represents a Test task that has finished. This event will eventually contain the finished status: success, error or failure. +This event represents a finished testCase execution. The event will contain the finished status and additional metadata as applicable. -- Event Type: __`dev.cdevents.testcase.finished.0.1.0`__ +- Event Type: __`dev.cdevents.testcase.finished.0.1.1`__ - Predicate: finished - Subject: [`testCase`](#testcase) -| Field | Type | Description | Examples | Required | -|-------------|-----------------|---------------------------------------------------------------------------------|-------------------------------------------|----------| -| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | -| executionId | `String` | An optional execution ID to enable handling of multiple simultaneous executions | | | -| status | `String` | The status of the testSuite execution, one of `passed`, `failed`, `aborted` | | ✅ | -| reason | `String` | An optional reason related to the status of the execution | `Cancelled by user`, `Failed assertion` | | +| Field | Type | Description | Examples | Required | +|-------------|-----------------|-------------------------------------------------------------------------------------|-------------------------------------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| executionId | `String` | An optional execution ID to enable handling of multiple simultaneous executions | | | +| status | `String` | The status of the testSuite execution, one of `passed`, `failed`, `aborted` | | ✅ | +| severity | `String` | An optional severity if the test failed, one of `critical`, `low`, `medium`, `high` | `critical`, `low`, `medium`, `high` | +| reason | `String` | An optional reason related to the status of the execution | `Cancelled by user`, `Failed assertion` | | ### `testSuite started` -This event represents a Test suite that has been started. +This event represents a started testSuite execution. - Event Type: __`dev.cdevents.testsuite.started.0.1.0`__ - Predicate: started @@ -109,29 +109,30 @@ This event represents a Test suite that has been started. ### `testSuite finished` -This event represents a Test suite that has has finished, the event will contain the finished status: success, error or failure. +This event represents a finished testSuite execution. The event will contain the execution status and additional metadata as applicable. - Event Type: __`dev.cdevents.testsuite.finished.0.1.0`__ - Predicate: finished - Subject: [`testSuite`](#testsuite) -| Field | Type | Description | Examples | Required | -|--------|-----------------|-----------------------------------------------------------------------------|----------------------------------------|----------| -| id | `String` | Uniquely identifies the subject within the source. | `unit`, `e2e`, `security` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | -| status | `String` | The status of the testSuite execution, one of `passed`, `failed`, `aborted` | | ✅ | -| reason | `String` | An optional reason related to the status of the execution | `Cancelled by user`, `Failed testCase` | | +| Field | Type | Description | Examples | Required | +|----------|-----------------|------------------------------------------------------------------------------------------|----------------------------------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `unit`, `e2e`, `security` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| status | `String` | The status of the testSuite execution, one of `passed`, `failed`, `aborted` | | ✅ | +| severity | `String` | An optional severity if the testSuite failed, one of `critical`, `low`, `medium`, `high` | `critical`, `low`, `medium`, `high` | +| reason | `String` | An optional reason related to the status of the execution | `Cancelled by user`, `Failed testCase` | | -### `testArtifact published` +### `testOutput published` -The event represents a test artifact that has been published and can be consumed by others. +The event represents a test execution output artifact that has been published. -- Event Type: __`dev.cdevents.testartifact.published.0.1.0`__ +- Event Type: __`dev.cdevents.testoutput.published.0.1.0`__ - Predicate: published -- Subject: [`testArtifact`](#testartifact) +- Subject: [`testOutput`](#testoutput) | Field | Type | Description | Examples | Required | |-------------|-----------------|--------------------------------------------------------------------------------------------------|------------|----------| -| id | `Purl` | Uniquely identifies the subject within the source. | `12312334` | ✅ | +| id | `String` | Uniquely identifies the subject within the source. | `12312334` | ✅ | | source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | | executionId | `String` | An optional execution ID to link this artifact to a specific `testCase` or `testSuite` execution | | | From 9ed418166990535fbb0c8bd1ff4a132cf1db6fbe Mon Sep 17 00:00:00 2001 From: olensmar Date: Wed, 18 Jan 2023 13:59:48 +0100 Subject: [PATCH 08/37] - added trigger metadata to testCase/testSuite started events --- testing-events.md | 26 +++++++++++++++----------- 1 file changed, 15 insertions(+), 11 deletions(-) diff --git a/testing-events.md b/testing-events.md index 603c96bb..0fc0a989 100644 --- a/testing-events.md +++ b/testing-events.md @@ -68,12 +68,14 @@ This event represents a started testCase execution. - Predicate: started - Subject: [`testCase`](#testcase) -| Field | Type | Description | Examples | Required | -|-------------|-----------------|--------------------------------------------------------------------------------------------|-------------------------------------------|----------| -| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | -| executionId | `String` | An optional execution ID to enable handling of multiple simultaneous executions | | | -| testSuiteId | `String` | An optional `testSuite` ID if this `testCase` was started as part of a testSuite execution | | | +| Field | Type | Description | Examples | Required | +|--------------|-----------------|--------------------------------------------------------------------------------------------------------------|-------------------------------------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| executionId | `String` | An optional execution ID to enable handling of multiple simultaneous executions | | | +| trigger.type | `String` | An optional cause of what triggered this testCase, one of `manual`, `pipeline`, `event`, `schedule`, `other` || +| trigger.uri | `URI-Reference` | An optional uri reference to what triggered this testCase || +| testSuiteId | `String` | An optional `testSuite` ID if this `testCase` was started as part of a testSuite execution | | | ### `testCase finished` @@ -101,11 +103,13 @@ This event represents a started testSuite execution. - Predicate: started - Subject: [`testSuite`](#testsuite) -| Field | Type | Description | Examples | Required | -|-------------|-----------------|---------------------------------------------------------------------------------|---------------------------|----------| -| id | `String` | Uniquely identifies the subject within the source. | `unit`, `e2e`, `security` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | -| executionId | `String` | An optional execution ID to enable handling of multiple simultaneous executions | | | +| Field | Type | Description | Examples | Required | +|--------------|-----------------|---------------------------------------------------------------------------------------------------------------|---------------------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `unit`, `e2e`, `security` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| executionId | `String` | An optional execution ID to enable handling of multiple simultaneous executions | | | +| trigger.type | `String` | An optional cause of what triggered this testSuite, one of `manual`, `pipeline`, `event`, `schedule`, `other` || +| trigger.uri | `URI-Reference` | An optional uri reference to what triggered this testSuite || ### `testSuite finished` From 950efea8fc847c18da86bbb800bad44f9d586ff2 Mon Sep 17 00:00:00 2001 From: olensmar Date: Thu, 19 Jan 2023 09:40:49 +0100 Subject: [PATCH 09/37] - fixed refs, bumped version of test events --- spec.md | 40 ++++++++++++++++++++-------------------- testing-events.md | 33 ++++++++++++++++----------------- 2 files changed, 36 insertions(+), 37 deletions(-) diff --git a/spec.md b/spec.md index 6a7d5ceb..cad64177 100644 --- a/spec.md +++ b/spec.md @@ -27,19 +27,19 @@ CDEvents is a common specification for Continuous Delivery events. - [Types](#types) - [CDEvent context](#cdevent-context) - [REQUIRED Context Attributes](#required-context-attributes) - - [id (context)](#id-context) - - [type (context)](#type-context) - - [source (context)](#source-context) + - [id (context)](#id--context-) + - [type (context)](#type--context-) + - [source (context)](#source--context-) - [timestamp](#timestamp) - [version](#version) - [Context example](#context-example) - [CDEvent subject](#cdevent-subject) - [REQUIRED Subject Attributes](#required-subject-attributes) - - [id (subject)](#id-subject) + - [id (subject)](#id--subject-) - [content](#content) - [OPTIONAL Subject Attributes](#optional-subject-attributes) - - [source (subject)](#source-subject) - - [type (subject)](#type-subject) + - [source (subject)](#source--subject-) + - [type (subject)](#type--subject-) - [Subject example](#subject-example) - [CDEvents custom data](#cdevents-custom-data) - [OPTIONAL Custom Data attributes](#optional-custom-data-attributes) @@ -170,14 +170,14 @@ defined in the [vocabulary](#vocabulary): - Type: [`String`][typesystem] - Description: Identifier for an event. Subsequent delivery attempts of the same event MAY share the same - [`id`](#id-context). This attribute matches the syntax and semantics of the + [`id`](#id--context-). This attribute matches the syntax and semantics of the [`id`](https://github.com/cloudevents/spec/blob/v1.0.2/cloudevents/spec.md#id) attribute of CloudEvents. - Constraints: - REQUIRED - MUST be a non-empty string - - MUST be unique within the given [`source`](#source-context) (in the scope of + - MUST be unique within the given [`source`](#source--context-) (in the scope of the producer) - Examples: - A [UUID version 4](https://en.wikipedia.org/wiki/Universally_unique_identifier#Version_4_(random)) @@ -204,8 +204,8 @@ defined in the [vocabulary](#vocabulary): - Type: [`URI-Reference`][typesystem] - Description: defines the context in which an event happened. The main purpose - of the source is to provide global uniqueness for [`source`](#source-context) + - [`id`](#id-context). + of the source is to provide global uniqueness for [`source`](#source--context-) + + [`id`](#id--context-). The source MAY identify a single producer or a group of producer that belong to the same application. @@ -244,7 +244,7 @@ defined in the [vocabulary](#vocabulary): In case the transport layer should require a re-transmission of the event, the timestamp SHOULD NOT be updated, i.e. it should be the same for the same - [`source`](#source-context) + [`id`](#id-context) combination. + [`source`](#source--context-) + [`id`](#id--context-) combination. - Constraints: - REQUIRED @@ -290,12 +290,12 @@ defined in the [vocabulary](#vocabulary): - Type: [`String`][typesystem] - Description: Identifier for a subject. Subsequent events associated to the same subject MUST use the same subject - [`id`](#id-subject). + [`id`](#id--subject-). - Constraints: - REQUIRED - MUST be a non-empty string - - MUST be unique within the given [`source`](#source-subject) (in the scope of + - MUST be unique within the given [`source`](#source--subject-) (in the scope of the producer) - Examples: - A [UUID version 4](https://en.wikipedia.org/wiki/Universally_unique_identifier#Version_4_(random)) @@ -305,7 +305,7 @@ defined in the [vocabulary](#vocabulary): - Type: [`Object`](#types) - Description: This provides all the relevant details of the [`content`](#content). The format of the [`content`](#content) depends on the - event [`type`](#type-context). All attributes in the subject + event [`type`](#type--context-). All attributes in the subject [`content`](#content), REQUIRED and OPTIONAL ones, MUST comply with the specification from the [vocabulary](#vocabulary). The [`content`](#content) may be empty. @@ -330,13 +330,13 @@ defined in the [vocabulary](#vocabulary): - Type: [`URI-Reference`][typesystem] - Description: defines the context in which the subject originated. In most - cases the [`source`](#source-subject) of the subject matches the - [`source`](#source-context) of the event. This field should be used only in - cases where the [`source`](#source-subject) of the *subject* is different from + cases the [`source`](#source--subject-) of the subject matches the + [`source`](#source--context-) of the event. This field should be used only in + cases where the [`source`](#source--subject-) of the *subject* is different from the [`source`](#source-context) of the event. - The format and semantic of the *subject* [`source`](#source-subject) are the - same of those of the *context* [`source`](#source-context). + The format and semantic of the *subject* [`source`](#source--subject-) are the + same of those of the *context* [`source`](#source--context-). #### type (subject) @@ -516,4 +516,4 @@ platforms. [typesystem]: https://github.com/cloudevents/spec/blob/v1.0.2/cloudevents/spec.md#type-system [datacontenttype]: https://github.com/cloudevents/spec/blob/v1.0.2/cloudevents/spec.md#datacontenttype [rfc2406]: https://tools.ietf.org/html/rfc2046 -[purl-spec]: https://github.com/package-url/purl-spec/blob/master/PURL-SPECIFICATION.rst \ No newline at end of file +[purl-spec]: https://github.com/package-url/purl-spec/blob/master/PURL-SPECIFICATION.rst diff --git a/testing-events.md b/testing-events.md index 0fc0a989..3b6bbaa0 100644 --- a/testing-events.md +++ b/testing-events.md @@ -10,8 +10,7 @@ description: > --> # Testing Events -Testing events covers the subjects and predicates related to test-execution -performed independently of CI/CD pipelines. +Testing events covers the subjects and predicates related to test-execution performed either independently or as part of CI/CD pipelines. ## Subjects @@ -64,7 +63,7 @@ One or more `testOutput` artifacts are usually produced as the result of a test This event represents a started testCase execution. -- Event Type: __`dev.cdevents.testcase.started.0.1.1`__ +- Event Type: __`dev.cdevents.testcase.started.0.2.0-draft`__ - Predicate: started - Subject: [`testCase`](#testcase) @@ -72,34 +71,34 @@ This event represents a started testCase execution. |--------------|-----------------|--------------------------------------------------------------------------------------------------------------|-------------------------------------------|----------| | id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | | source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | -| executionId | `String` | An optional execution ID to enable handling of multiple simultaneous executions | | | | trigger.type | `String` | An optional cause of what triggered this testCase, one of `manual`, `pipeline`, `event`, `schedule`, `other` || | trigger.uri | `URI-Reference` | An optional uri reference to what triggered this testCase || | testSuiteId | `String` | An optional `testSuite` ID if this `testCase` was started as part of a testSuite execution | | | +| executionId | `String` | An optional execution ID to enable handling of multiple simultaneous executions of this testCase | | | ### `testCase finished` This event represents a finished testCase execution. The event will contain the finished status and additional metadata as applicable. -- Event Type: __`dev.cdevents.testcase.finished.0.1.1`__ +- Event Type: __`dev.cdevents.testcase.finished.0.2.0-draft`__ - Predicate: finished - Subject: [`testCase`](#testcase) -| Field | Type | Description | Examples | Required | -|-------------|-----------------|-------------------------------------------------------------------------------------|-------------------------------------------|----------| -| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | -| executionId | `String` | An optional execution ID to enable handling of multiple simultaneous executions | | | -| status | `String` | The status of the testSuite execution, one of `passed`, `failed`, `aborted` | | ✅ | -| severity | `String` | An optional severity if the test failed, one of `critical`, `low`, `medium`, `high` | `critical`, `low`, `medium`, `high` | -| reason | `String` | An optional reason related to the status of the execution | `Cancelled by user`, `Failed assertion` | | +| Field | Type | Description | Examples | Required | +|-------------|-----------------|---------------------------------------------------------------------------------------------------|-------------------------------------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| status | `String` | The status of the testSuite execution, one of `passed`, `failed`, `aborted` | | ✅ | +| severity | `String` | An optional severity if the test failed, one of `critical`, `low`, `medium`, `high` | `critical`, `low`, `medium`, `high` | +| reason | `String` | An optional reason related to the status of the execution | `Cancelled by user`, `Failed assertion` | | +| executionId | `String` | An optional execution ID to enable handling of multiple simultaneous executions of this testSuite | | | ### `testSuite started` This event represents a started testSuite execution. -- Event Type: __`dev.cdevents.testsuite.started.0.1.0`__ +- Event Type: __`dev.cdevents.testsuite.started.0.2.0-draft`__ - Predicate: started - Subject: [`testSuite`](#testsuite) @@ -108,8 +107,8 @@ This event represents a started testSuite execution. | id | `String` | Uniquely identifies the subject within the source. | `unit`, `e2e`, `security` | ✅ | | source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | | executionId | `String` | An optional execution ID to enable handling of multiple simultaneous executions | | | -| trigger.type | `String` | An optional cause of what triggered this testSuite, one of `manual`, `pipeline`, `event`, `schedule`, `other` || -| trigger.uri | `URI-Reference` | An optional uri reference to what triggered this testSuite || +| trigger.type | `String` | An optional cause of what triggered this testSuite, one of `manual`, `pipeline`, `event`, `schedule`, `other` ||| +| trigger.uri | `URI-Reference` | An optional uri reference to what triggered this testSuite ||| ### `testSuite finished` @@ -131,7 +130,7 @@ This event represents a finished testSuite execution. The event will contain the The event represents a test execution output artifact that has been published. -- Event Type: __`dev.cdevents.testoutput.published.0.1.0`__ +- Event Type: __`dev.cdevents.testoutput.published.0.2.0-draft`__ - Predicate: published - Subject: [`testOutput`](#testoutput) From 284458b82653e967cea30e812032586dbc5616e9 Mon Sep 17 00:00:00 2001 From: olensmar Date: Thu, 19 Jan 2023 09:44:42 +0100 Subject: [PATCH 10/37] - fixed links to primer --- spec.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/spec.md b/spec.md index cad64177..b759f4b2 100644 --- a/spec.md +++ b/spec.md @@ -333,7 +333,7 @@ defined in the [vocabulary](#vocabulary): cases the [`source`](#source--subject-) of the subject matches the [`source`](#source--context-) of the event. This field should be used only in cases where the [`source`](#source--subject-) of the *subject* is different from - the [`source`](#source-context) of the event. + the [`source`](#source--context-) of the event. The format and semantic of the *subject* [`source`](#source--subject-) are the same of those of the *context* [`source`](#source--context-). From 141d60874938f9b7383eac20c95c00d7125d55fe Mon Sep 17 00:00:00 2001 From: olensmar Date: Thu, 19 Jan 2023 09:52:31 +0100 Subject: [PATCH 11/37] - added environment to testCase/testSuite started events and removed testSuiteId from testCase started --- testing-events.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/testing-events.md b/testing-events.md index 3b6bbaa0..cf4f6fe4 100644 --- a/testing-events.md +++ b/testing-events.md @@ -73,7 +73,7 @@ This event represents a started testCase execution. | source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | | trigger.type | `String` | An optional cause of what triggered this testCase, one of `manual`, `pipeline`, `event`, `schedule`, `other` || | trigger.uri | `URI-Reference` | An optional uri reference to what triggered this testCase || -| testSuiteId | `String` | An optional `testSuite` ID if this `testCase` was started as part of a testSuite execution | | | +| environment | `String` | An optional identifier for the environment in which this testCase is executing | `dev`, `prod` | | executionId | `String` | An optional execution ID to enable handling of multiple simultaneous executions of this testCase | | | ### `testCase finished` @@ -109,6 +109,7 @@ This event represents a started testSuite execution. | executionId | `String` | An optional execution ID to enable handling of multiple simultaneous executions | | | | trigger.type | `String` | An optional cause of what triggered this testSuite, one of `manual`, `pipeline`, `event`, `schedule`, `other` ||| | trigger.uri | `URI-Reference` | An optional uri reference to what triggered this testSuite ||| +| environment | `String` | An optional identifier for the environment in which this testSuite is executing | `dev`, `prod` | ### `testSuite finished` From 32da41daa7b43e0d21cdd2f03617fdd162a860f1 Mon Sep 17 00:00:00 2001 From: olensmar Date: Thu, 19 Jan 2023 10:59:55 +0100 Subject: [PATCH 12/37] - added schemas and aligned vocabulary --- schemas/testcasefinished.json | 54 +++++++++----- schemas/testcasestarted.json | 70 +++++++++++++++---- ...sequeued.json => testoutputpublished.json} | 55 ++++++++++----- schemas/testsuitefinished.json | 56 +++++++++++---- schemas/testsuitestarted.json | 57 +++++++++++---- testing-events.md | 34 ++++----- 6 files changed, 232 insertions(+), 94 deletions(-) rename schemas/{testcasequeued.json => testoutputpublished.json} (54%) diff --git a/schemas/testcasefinished.json b/schemas/testcasefinished.json index 38763749..5ce54b27 100644 --- a/schemas/testcasefinished.json +++ b/schemas/testcasefinished.json @@ -1,6 +1,6 @@ { "$schema": "https://json-schema.org/draft/2020-12/schema", - "$id": "https://cdevents.dev/0.2.0/schema/test-case-finished-event", + "$id": "https://cdevents.dev/0.2.0-draft/schema/test-case-finished-event", "properties": { "context": { "properties": { @@ -14,15 +14,14 @@ }, "source": { "type": "string", - "minLength": 1, - "format": "uri-reference" + "minLength": 1 }, "type": { "type": "string", "enum": [ - "dev.cdevents.testcase.finished.0.1.1" + "dev.cdevents.testcase.finished.0.2.0-draft" ], - "default": "dev.cdevents.testcase.finished.0.1.1" + "default": "dev.cdevents.testcase.finished.0.2.0-draft" }, "timestamp": { "type": "string", @@ -46,22 +45,45 @@ "minLength": 1 }, "source": { - "type": "string", - "minLength": 1, - "format": "uri-reference" + "type": "string" }, "type": { "type": "string", - "minLength": 1, - "enum": [ - "testCase" - ], - "default": "testCase" + "minLength": 1 }, "content": { - "properties": {}, + "properties": { + "status": { + "type": "string", + "enum": [ + "pass", + "fail", + "aborted" + ] + }, + "severity": { + "type": "string", + "enum": [ + "critical", + "low", + "medium", + "high" + ] + }, + "reason": { + "type": "string", + "description": "an optional reason to why the testCase ended as it did" + }, + "executionId": { + "type": "string", + "description": "an optional identifier of the testCase execution" + } + }, "additionalProperties": false, - "type": "object" + "type": "object", + "required": [ + "status" + ] } }, "additionalProperties": false, @@ -93,4 +115,4 @@ "context", "subject" ] -} \ No newline at end of file +} diff --git a/schemas/testcasestarted.json b/schemas/testcasestarted.json index 37114881..50ea0523 100644 --- a/schemas/testcasestarted.json +++ b/schemas/testcasestarted.json @@ -1,6 +1,6 @@ { "$schema": "https://json-schema.org/draft/2020-12/schema", - "$id": "https://cdevents.dev/0.2.0/schema/test-case-started-event", + "$id": "https://cdevents.dev/0.2.0-draft/schema/test-case-started-event", "properties": { "context": { "properties": { @@ -14,15 +14,14 @@ }, "source": { "type": "string", - "minLength": 1, - "format": "uri-reference" + "minLength": 1 }, "type": { "type": "string", "enum": [ - "dev.cdevents.testcase.started.0.1.1" + "dev.cdevents.testcase.started.0.2.0-draft" ], - "default": "dev.cdevents.testcase.started.0.1.1" + "default": "dev.cdevents.testcase.started.0.2.0-draft" }, "timestamp": { "type": "string", @@ -46,20 +45,61 @@ "minLength": 1 }, "source": { - "type": "string", - "minLength": 1, - "format": "uri-reference" + "type": "string" }, "type": { "type": "string", - "minLength": 1, - "enum": [ - "testCase" - ], - "default": "testCase" + "minLength": 1 }, "content": { - "properties": {}, + "properties": { + "type": { + "type": "string", + "enum": [ + "load", + "functional", + "unit", + "security", + "compliance", + "integration", + "e2e", + "other" + ] + }, + "uri": { + "type": "string", + "format": "uri", + "description": "an optional URI to the actual testCase" + }, + "trigger": { + "type": "object", + "properties": { + "type": { + "type": "string", + "enum": [ + "manual", + "pipeline", + "event", + "schedule", + "other" + ] + }, + "uri": { + "type": "string", + "format": "uri" + }, + "description": "an optional description of what triggered this testCase execution" + } + }, + "environment": { + "type": "string", + "description": "an optional identifier of the environment in which this testCase is running" + }, + "executionId": { + "type": "string", + "description": "an optional identifier of the testCase execution" + } + }, "additionalProperties": false, "type": "object" } @@ -93,4 +133,4 @@ "context", "subject" ] -} \ No newline at end of file +} diff --git a/schemas/testcasequeued.json b/schemas/testoutputpublished.json similarity index 54% rename from schemas/testcasequeued.json rename to schemas/testoutputpublished.json index b2a7589b..ebabbcd6 100644 --- a/schemas/testcasequeued.json +++ b/schemas/testoutputpublished.json @@ -1,6 +1,6 @@ { "$schema": "https://json-schema.org/draft/2020-12/schema", - "$id": "https://cdevents.dev/0.2.0/schema/test-case-queued-event", + "$id": "https://cdevents.dev/0.2.0-draft/schema/test-output-published-event", "properties": { "context": { "properties": { @@ -14,15 +14,14 @@ }, "source": { "type": "string", - "minLength": 1, - "format": "uri-reference" + "minLength": 1 }, "type": { "type": "string", "enum": [ - "dev.cdevents.testcase.queued.0.1.1" + "dev.cdevents.testoutput.published.0.2.0-draft" ], - "default": "dev.cdevents.testcase.queued.0.1.1" + "default": "dev.cdevents.testoutput.published.0.2.0-draft" }, "timestamp": { "type": "string", @@ -46,22 +45,46 @@ "minLength": 1 }, "source": { - "type": "string", - "minLength": 1, - "format": "uri-reference" + "type": "string" }, "type": { "type": "string", - "minLength": 1, - "enum": [ - "testCase" - ], - "default": "testCase" + "minLength": 1 }, "content": { - "properties": {}, + "properties": { + "type": { + "type": "string", + "enum": [ + "report", + "video", + "image", + "log", + "other" + ] + }, + "format": { + "type": "string", + "description": "The content-type of the output artifact", + "example": "application/pdf" + }, + "uri": { + "type": "string", + "format": "uri", + "description": "an optional URI to the actual testOutput artifact" + }, + "executionId": { + "type": "string", + "description": "an optional identifier of the testCase or testSuite execution that produced this output" + } + }, "additionalProperties": false, - "type": "object" + "type": "object", + "required": [ + "type", + "format", + "uri" + ] } }, "additionalProperties": false, @@ -93,4 +116,4 @@ "context", "subject" ] -} \ No newline at end of file +} diff --git a/schemas/testsuitefinished.json b/schemas/testsuitefinished.json index 847650b2..b932405c 100644 --- a/schemas/testsuitefinished.json +++ b/schemas/testsuitefinished.json @@ -1,6 +1,6 @@ { "$schema": "https://json-schema.org/draft/2020-12/schema", - "$id": "https://cdevents.dev/0.2.0/schema/test-suite-finished-event", + "$id": "https://cdevents.dev/0.2.0-draft/schema/test-suite-finished-event", "properties": { "context": { "properties": { @@ -14,15 +14,14 @@ }, "source": { "type": "string", - "minLength": 1, - "format": "uri-reference" + "minLength": 1 }, "type": { "type": "string", "enum": [ - "dev.cdevents.testsuite.finished.0.1.1" + "dev.cdevents.testsuite.finished.0.2.0-draft" ], - "default": "dev.cdevents.testsuite.finished.0.1.1" + "default": "dev.cdevents.testsuite.finished.0.2.0-draft" }, "timestamp": { "type": "string", @@ -46,20 +45,47 @@ "minLength": 1 }, "source": { - "type": "string", - "minLength": 1, - "format": "uri-reference" + "type": "string" }, "type": { "type": "string", - "minLength": 1, - "enum": [ - "testSuite" - ], - "default": "testSuite" + "minLength": 1 }, "content": { - "properties": {}, + "properties": { + "properties": { + "status": { + "type": "string", + "enum": [ + "pass", + "fail", + "aborted" + ] + }, + "severity": { + "type": "string", + "enum": [ + "critical", + "low", + "medium", + "high" + ] + }, + "reason": { + "type": "string", + "description": "an optional reason to why the testSuite ended as it did" + }, + "executionId": { + "type": "string", + "description": "an optional identifier of the testSuite execution" + } + }, + "additionalProperties": false, + "type": "object", + "required": [ + "status" + ] + }, "additionalProperties": false, "type": "object" } @@ -93,4 +119,4 @@ "context", "subject" ] -} \ No newline at end of file +} diff --git a/schemas/testsuitestarted.json b/schemas/testsuitestarted.json index 1e388efc..cb248446 100644 --- a/schemas/testsuitestarted.json +++ b/schemas/testsuitestarted.json @@ -1,6 +1,6 @@ { "$schema": "https://json-schema.org/draft/2020-12/schema", - "$id": "https://cdevents.dev/0.2.0/schema/test-suite-started-event", + "$id": "https://cdevents.dev/0.2.0-draft/schema/test-suite-started-event", "properties": { "context": { "properties": { @@ -14,15 +14,14 @@ }, "source": { "type": "string", - "minLength": 1, - "format": "uri-reference" + "minLength": 1 }, "type": { "type": "string", "enum": [ - "dev.cdevents.testsuite.started.0.1.1" + "dev.cdevents.testsuite.started.0.2.0-draft" ], - "default": "dev.cdevents.testsuite.started.0.1.1" + "default": "dev.cdevents.testsuite.started.0.2.0-draft" }, "timestamp": { "type": "string", @@ -46,20 +45,48 @@ "minLength": 1 }, "source": { - "type": "string", - "minLength": 1, - "format": "uri-reference" + "type": "string" }, "type": { "type": "string", - "minLength": 1, - "enum": [ - "testSuite" - ], - "default": "testSuite" + "minLength": 1 }, "content": { - "properties": {}, + "properties": { + "uri": { + "type": "string", + "format": "uri", + "description": "an optional URI to the actual testSuite" + }, + "trigger": { + "type": "object", + "properties": { + "type": { + "type": "string", + "enum": [ + "manual", + "pipeline", + "event", + "schedule", + "other" + ] + }, + "uri": { + "type": "string", + "format": "uri" + }, + "description": "an optional description of what triggered this testSuite execution" + } + }, + "environment": { + "type": "string", + "description": "an optional identifier of the environment in which this testSuite is running" + }, + "executionId": { + "type": "string", + "description": "an optional identifier of the testSuite execution" + } + }, "additionalProperties": false, "type": "object" } @@ -93,4 +120,4 @@ "context", "subject" ] -} \ No newline at end of file +} diff --git a/testing-events.md b/testing-events.md index cf4f6fe4..a9297ff2 100644 --- a/testing-events.md +++ b/testing-events.md @@ -54,8 +54,7 @@ One or more `testOutput` artifacts are usually produced as the result of a test | source | `URI-Reference` | [source](spec.md#source--context-) from the context | `staging/tekton`, `tekton-dev-123` | | type | `String` | The type of output, one of `report`, `video`, `image`, `log`, `other` | `video` | | format | `String` | The Content-Type of the output artifact | `application/pdf`, `image/png`, `application/json` | -| url | `URI-Reference` | A reference to retrieve the specified output artifact | `https://testkube.mycluster.internal/artifacts/23123123` | -| testCaseId | `URI-Reference` | An optional reference to the testCase resulting in this output artifact | `https://testkube.mycluster.internal/testCase/123123` | +| uri | `URI-Reference` | A reference to retrieve the specified output artifact | `https://testkube.mycluster.internal/artifacts/23123123` | ## Events @@ -84,14 +83,14 @@ This event represents a finished testCase execution. The event will contain the - Predicate: finished - Subject: [`testCase`](#testcase) -| Field | Type | Description | Examples | Required | -|-------------|-----------------|---------------------------------------------------------------------------------------------------|-------------------------------------------|----------| -| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | -| status | `String` | The status of the testSuite execution, one of `passed`, `failed`, `aborted` | | ✅ | -| severity | `String` | An optional severity if the test failed, one of `critical`, `low`, `medium`, `high` | `critical`, `low`, `medium`, `high` | -| reason | `String` | An optional reason related to the status of the execution | `Cancelled by user`, `Failed assertion` | | -| executionId | `String` | An optional execution ID to enable handling of multiple simultaneous executions of this testSuite | | | +| Field | Type | Description | Examples | Required | +|-------------|-----------------|--------------------------------------------------------------------------------------------------|-------------------------------------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| status | `String` | The status of the testSuite execution, one of `passed`, `failed`, `aborted` | | ✅ | +| severity | `String` | An optional severity if the test failed, one of `critical`, `low`, `medium`, `high` | `critical`, `low`, `medium`, `high` | +| reason | `String` | An optional reason related to the status of the execution | `Cancelled by user`, `Failed assertion` | | +| executionId | `String` | An optional execution ID to enable handling of multiple simultaneous executions of this testCase | | | ### `testSuite started` @@ -119,13 +118,14 @@ This event represents a finished testSuite execution. The event will contain the - Predicate: finished - Subject: [`testSuite`](#testsuite) -| Field | Type | Description | Examples | Required | -|----------|-----------------|------------------------------------------------------------------------------------------|----------------------------------------|----------| -| id | `String` | Uniquely identifies the subject within the source. | `unit`, `e2e`, `security` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | -| status | `String` | The status of the testSuite execution, one of `passed`, `failed`, `aborted` | | ✅ | -| severity | `String` | An optional severity if the testSuite failed, one of `critical`, `low`, `medium`, `high` | `critical`, `low`, `medium`, `high` | -| reason | `String` | An optional reason related to the status of the execution | `Cancelled by user`, `Failed testCase` | | +| Field | Type | Description | Examples | Required | +|-------------|-----------------|---------------------------------------------------------------------------------------------------|----------------------------------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `unit`, `e2e`, `security` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| status | `String` | The status of the testSuite execution, one of `passed`, `failed`, `aborted` | | ✅ | +| severity | `String` | An optional severity if the testSuite failed, one of `critical`, `low`, `medium`, `high` | `critical`, `low`, `medium`, `high` | +| reason | `String` | An optional reason related to the status of the execution | `Cancelled by user`, `Failed testCase` | | +| executionId | `String` | An optional execution ID to enable handling of multiple simultaneous executions of this testSuite | | | ### `testOutput published` From 3c58cd5b3d15a0fbb7afa5faf8291647771b037b Mon Sep 17 00:00:00 2001 From: olensmar Date: Tue, 31 Jan 2023 10:22:47 +0100 Subject: [PATCH 13/37] updates in line with meeting discussion/feedback: - changed to testSuiteRun/testCaseRun - added queued events - changed version to 0.1.0-draft - introduces objects for environment and trigger - added not on deprecation of old subjects - updated schemas accordingly --- ...finished.json => testcaserunfinished.json} | 45 +++- schemas/testcaserunqueued.json | 160 ++++++++++++++ schemas/testcaserunstarted.json | 160 ++++++++++++++ schemas/testoutputpublished.json | 6 +- ...inished.json => testsuiterunfinished.json} | 13 +- ...testarted.json => testsuiterunqueued.json} | 34 ++- ...estarted.json => testsuiterunstarted.json} | 51 +++-- testing-events.md | 195 +++++++++++------- 8 files changed, 540 insertions(+), 124 deletions(-) rename schemas/{testcasefinished.json => testcaserunfinished.json} (63%) create mode 100644 schemas/testcaserunqueued.json create mode 100644 schemas/testcaserunstarted.json rename schemas/{testsuitefinished.json => testsuiterunfinished.json} (89%) rename schemas/{testsuitestarted.json => testsuiterunqueued.json} (72%) rename schemas/{testcasestarted.json => testsuiterunstarted.json} (72%) diff --git a/schemas/testcasefinished.json b/schemas/testcaserunfinished.json similarity index 63% rename from schemas/testcasefinished.json rename to schemas/testcaserunfinished.json index 5ce54b27..c0cffe3f 100644 --- a/schemas/testcasefinished.json +++ b/schemas/testcaserunfinished.json @@ -1,6 +1,6 @@ { "$schema": "https://json-schema.org/draft/2020-12/schema", - "$id": "https://cdevents.dev/0.2.0-draft/schema/test-case-finished-event", + "$id": "https://cdevents.dev/0.1.0-draft/schema/test-case-run-finished-event", "properties": { "context": { "properties": { @@ -19,9 +19,9 @@ "type": { "type": "string", "enum": [ - "dev.cdevents.testcase.finished.0.2.0-draft" + "dev.cdevents.testcaserun.finished.0.1.0-draft" ], - "default": "dev.cdevents.testcase.finished.0.2.0-draft" + "default": "dev.cdevents.testcaserun.finished.0.1.0-draft" }, "timestamp": { "type": "string", @@ -58,25 +58,54 @@ "enum": [ "pass", "fail", - "aborted" + "abort", + "error" ] }, "severity": { "type": "string", "enum": [ - "critical", "low", "medium", - "high" + "high", + "critical" ] }, "reason": { "type": "string", "description": "an optional reason to why the testCase ended as it did" }, - "executionId": { + "environment": { + "description": "the environment in which this testCaseRun is being queued", + "properties": { + "id": { + "type": "string", + "minLength": 1 + }, + "source": { + "type": "string" + }, + "name": { + "type": "string" + }, + "url": { + "type": "string", + "format": "uri" + } + }, + "additionalProperties": false, + "type": "object", + "required": [ + "id" + ] + }, + "testSuiteRunId": { + "type": "string", + "description": "an optional identifier of the containing testSuiteRun" + }, + "testCaseId": { "type": "string", - "description": "an optional identifier of the testCase execution" + "description": "an optional identifier of the queued testCase" } }, "additionalProperties": false, diff --git a/schemas/testcaserunqueued.json b/schemas/testcaserunqueued.json new file mode 100644 index 00000000..38268c0a --- /dev/null +++ b/schemas/testcaserunqueued.json @@ -0,0 +1,160 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://cdevents.dev/0.1.0-draft/schema/test-case-run-queued-event", + "properties": { + "context": { + "properties": { + "version": { + "type": "string", + "minLength": 1 + }, + "id": { + "type": "string", + "minLength": 1 + }, + "source": { + "type": "string", + "minLength": 1 + }, + "type": { + "type": "string", + "enum": [ + "dev.cdevents.testcaserun.queued.0.1.0-draft" + ], + "default": "dev.cdevents.testcaserun.queued.0.1.0-draft" + }, + "timestamp": { + "type": "string", + "format": "date-time" + } + }, + "additionalProperties": false, + "type": "object", + "required": [ + "version", + "id", + "source", + "type", + "timestamp" + ] + }, + "subject": { + "properties": { + "id": { + "type": "string", + "minLength": 1 + }, + "source": { + "type": "string" + }, + "type": { + "type": "string", + "minLength": 1 + }, + "content": { + "properties": { + "type": { + "type": "string", + "enum": [ + "performance", + "functional", + "unit", + "security", + "compliance", + "integration", + "e2e", + "other" + ] + }, + "uri": { + "type": "string", + "format": "uri", + "description": "an optional URI to the actual testCase" + }, + "trigger": { + "type": "object", + "description": "an optional definition of what triggered this testCase queueing", + "properties": { + "type": { + "type": "string", + "enum": [ + "manual", + "pipeline", + "event", + "schedule", + "other" + ] + }, + "uri": { + "type": "string", + "format": "uri" + } + } + }, + "environment": { + "description": "the environment in which this testCaseRun is being queued", + "properties": { + "id": { + "type": "string", + "minLength": 1 + }, + "source": { + "type": "string" + }, + "name": { + "type": "string" + }, + "url": { + "type": "string", + "format": "uri" + } + }, + "additionalProperties": false, + "type": "object", + "required": [ + "id" + ] + }, + "testSuiteRunId": { + "type": "string", + "description": "an optional identifier of the containing testSuiteRun" + }, + "testCaseId": { + "type": "string", + "description": "an optional identifier of the queued testCase" + } + }, + "additionalProperties": false, + "type": "object" + } + }, + "additionalProperties": false, + "type": "object", + "required": [ + "id", + "type", + "content" + ] + }, + "customData": { + "oneOf": [ + { + "type": "object" + }, + { + "type": "string", + "contentEncoding": "base64" + } + ] + }, + "customDataContentType": { + "type": "string" + } + }, + "additionalProperties": false, + "type": "object", + "required": [ + "context", + "subject" + ] +} diff --git a/schemas/testcaserunstarted.json b/schemas/testcaserunstarted.json new file mode 100644 index 00000000..d263684e --- /dev/null +++ b/schemas/testcaserunstarted.json @@ -0,0 +1,160 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://cdevents.dev/0.1.0-draft/schema/test-case-run-started-event", + "properties": { + "context": { + "properties": { + "version": { + "type": "string", + "minLength": 1 + }, + "id": { + "type": "string", + "minLength": 1 + }, + "source": { + "type": "string", + "minLength": 1 + }, + "type": { + "type": "string", + "enum": [ + "dev.cdevents.testcaserun.started.0.1.0-draft" + ], + "default": "dev.cdevents.testcaserun.started.0.1.0-draft" + }, + "timestamp": { + "type": "string", + "format": "date-time" + } + }, + "additionalProperties": false, + "type": "object", + "required": [ + "version", + "id", + "source", + "type", + "timestamp" + ] + }, + "subject": { + "properties": { + "id": { + "type": "string", + "minLength": 1 + }, + "source": { + "type": "string" + }, + "type": { + "type": "string", + "minLength": 1 + }, + "content": { + "properties": { + "type": { + "type": "string", + "enum": [ + "performance", + "functional", + "unit", + "security", + "compliance", + "integration", + "e2e", + "other" + ] + }, + "uri": { + "type": "string", + "format": "uri", + "description": "an optional URI to the actual testCase" + }, + "trigger": { + "type": "object", + "description": "an optional definition of what triggered this testCase execution", + "properties": { + "type": { + "type": "string", + "enum": [ + "manual", + "pipeline", + "event", + "schedule", + "other" + ] + }, + "uri": { + "type": "string", + "format": "uri" + } + } + }, + "environment": { + "description": "the environment in which this testCaseRun started", + "properties": { + "id": { + "type": "string", + "minLength": 1 + }, + "source": { + "type": "string" + }, + "name": { + "type": "string" + }, + "url": { + "type": "string", + "format": "uri" + } + }, + "additionalProperties": false, + "type": "object", + "required": [ + "id" + ] + }, + "testSuiteRunId": { + "type": "string", + "description": "an optional identifier of the containing testSuiteRun" + }, + "testCaseId": { + "type": "string", + "description": "an optional identifier of the running testCase" + } + }, + "additionalProperties": false, + "type": "object" + } + }, + "additionalProperties": false, + "type": "object", + "required": [ + "id", + "type", + "content" + ] + }, + "customData": { + "oneOf": [ + { + "type": "object" + }, + { + "type": "string", + "contentEncoding": "base64" + } + ] + }, + "customDataContentType": { + "type": "string" + } + }, + "additionalProperties": false, + "type": "object", + "required": [ + "context", + "subject" + ] +} diff --git a/schemas/testoutputpublished.json b/schemas/testoutputpublished.json index ebabbcd6..c3a0877b 100644 --- a/schemas/testoutputpublished.json +++ b/schemas/testoutputpublished.json @@ -1,6 +1,6 @@ { "$schema": "https://json-schema.org/draft/2020-12/schema", - "$id": "https://cdevents.dev/0.2.0-draft/schema/test-output-published-event", + "$id": "https://cdevents.dev/0.1.0-draft/schema/test-output-published-event", "properties": { "context": { "properties": { @@ -19,9 +19,9 @@ "type": { "type": "string", "enum": [ - "dev.cdevents.testoutput.published.0.2.0-draft" + "dev.cdevents.testoutput.published.0.1.0-draft" ], - "default": "dev.cdevents.testoutput.published.0.2.0-draft" + "default": "dev.cdevents.testoutput.published.0.1.0-draft" }, "timestamp": { "type": "string", diff --git a/schemas/testsuitefinished.json b/schemas/testsuiterunfinished.json similarity index 89% rename from schemas/testsuitefinished.json rename to schemas/testsuiterunfinished.json index b932405c..62ff9ad1 100644 --- a/schemas/testsuitefinished.json +++ b/schemas/testsuiterunfinished.json @@ -1,6 +1,6 @@ { "$schema": "https://json-schema.org/draft/2020-12/schema", - "$id": "https://cdevents.dev/0.2.0-draft/schema/test-suite-finished-event", + "$id": "https://cdevents.dev/0.1.0-draft/schema/test-suite-finished-event", "properties": { "context": { "properties": { @@ -19,9 +19,9 @@ "type": { "type": "string", "enum": [ - "dev.cdevents.testsuite.finished.0.2.0-draft" + "dev.cdevents.testsuiterun.finished.0.1.0-draft" ], - "default": "dev.cdevents.testsuite.finished.0.2.0-draft" + "default": "dev.cdevents.testsuiterun.finished.0.1.0-draft" }, "timestamp": { "type": "string", @@ -59,16 +59,17 @@ "enum": [ "pass", "fail", - "aborted" + "abort", + "error" ] }, "severity": { "type": "string", "enum": [ - "critical", "low", "medium", - "high" + "high", + "critical" ] }, "reason": { diff --git a/schemas/testsuitestarted.json b/schemas/testsuiterunqueued.json similarity index 72% rename from schemas/testsuitestarted.json rename to schemas/testsuiterunqueued.json index cb248446..6299406d 100644 --- a/schemas/testsuitestarted.json +++ b/schemas/testsuiterunqueued.json @@ -1,6 +1,6 @@ { "$schema": "https://json-schema.org/draft/2020-12/schema", - "$id": "https://cdevents.dev/0.2.0-draft/schema/test-suite-started-event", + "$id": "https://cdevents.dev/0.1.0-draft/schema/test-suite-run-queued-event", "properties": { "context": { "properties": { @@ -19,9 +19,9 @@ "type": { "type": "string", "enum": [ - "dev.cdevents.testsuite.started.0.2.0-draft" + "dev.cdevents.testsuiterun.queued.0.1.0-draft" ], - "default": "dev.cdevents.testsuite.started.0.2.0-draft" + "default": "dev.cdevents.testsuiterun.queued.0.1.0-draft" }, "timestamp": { "type": "string", @@ -60,6 +60,7 @@ }, "trigger": { "type": "object", + "description": "an optional definition of what triggered this testCase execution", "properties": { "type": { "type": "string", @@ -74,13 +75,32 @@ "uri": { "type": "string", "format": "uri" - }, - "description": "an optional description of what triggered this testSuite execution" + } } }, "environment": { - "type": "string", - "description": "an optional identifier of the environment in which this testSuite is running" + "description": "the environment in which this testSuiteRun started", + "properties": { + "id": { + "type": "string", + "minLength": 1 + }, + "source": { + "type": "string" + }, + "name": { + "type": "string" + }, + "url": { + "type": "string", + "format": "uri" + } + }, + "additionalProperties": false, + "type": "object", + "required": [ + "id" + ] }, "executionId": { "type": "string", diff --git a/schemas/testcasestarted.json b/schemas/testsuiterunstarted.json similarity index 72% rename from schemas/testcasestarted.json rename to schemas/testsuiterunstarted.json index 50ea0523..3d42c46d 100644 --- a/schemas/testcasestarted.json +++ b/schemas/testsuiterunstarted.json @@ -1,6 +1,6 @@ { "$schema": "https://json-schema.org/draft/2020-12/schema", - "$id": "https://cdevents.dev/0.2.0-draft/schema/test-case-started-event", + "$id": "https://cdevents.dev/0.1.0-draft/schema/test-suite-run-started-event", "properties": { "context": { "properties": { @@ -19,9 +19,9 @@ "type": { "type": "string", "enum": [ - "dev.cdevents.testcase.started.0.2.0-draft" + "dev.cdevents.testsuiterun.started.0.1.0-draft" ], - "default": "dev.cdevents.testcase.started.0.2.0-draft" + "default": "dev.cdevents.testsuiterun.started.0.1.0-draft" }, "timestamp": { "type": "string", @@ -53,26 +53,14 @@ }, "content": { "properties": { - "type": { - "type": "string", - "enum": [ - "load", - "functional", - "unit", - "security", - "compliance", - "integration", - "e2e", - "other" - ] - }, "uri": { "type": "string", "format": "uri", - "description": "an optional URI to the actual testCase" + "description": "an optional URI to the actual testSuite" }, "trigger": { "type": "object", + "description": "an optional definition of what triggered this testCase execution", "properties": { "type": { "type": "string", @@ -87,17 +75,36 @@ "uri": { "type": "string", "format": "uri" - }, - "description": "an optional description of what triggered this testCase execution" + } } }, "environment": { - "type": "string", - "description": "an optional identifier of the environment in which this testCase is running" + "description": "the environment in which this testSuiteRun started", + "properties": { + "id": { + "type": "string", + "minLength": 1 + }, + "source": { + "type": "string" + }, + "name": { + "type": "string" + }, + "url": { + "type": "string", + "format": "uri" + } + }, + "additionalProperties": false, + "type": "object", + "required": [ + "id" + ] }, "executionId": { "type": "string", - "description": "an optional identifier of the testCase execution" + "description": "an optional identifier of the testSuite execution" } }, "additionalProperties": false, diff --git a/testing-events.md b/testing-events.md index a9297ff2..24572601 100644 --- a/testing-events.md +++ b/testing-events.md @@ -12,131 +12,170 @@ description: > Testing events covers the subjects and predicates related to test-execution performed either independently or as part of CI/CD pipelines. +> These subjects replace the previously defined testCase/testSuite subjects in the Continuous Integration category + ## Subjects This specification defines three subjects in this stage: `testCase`, `testSuite` and `testArtifact`. -| Subject | Description | Predicates | -|-----------------------------|-------------------------------------------------|--------------------------------------------------------------------| -| [`testCase`](#testcase) | A software test case | [`started`](#testcase-started), [`finished`](#testcase-finished) | -| [`testSuite`](#testsuite) | A collection of test cases | [`started`](#testsuite-started), [`finished`](#testsuite-finished) | -| [`testOutput`](#testoutput) | An output artifact produced by a test execution | [`published`](#testoutput-published) | +| Subject | Description | Predicates | +|---------------------------------|-------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------| +| [`testCaseRun`](#testcaserun) | The execution of a software testCase | [`queued`](#testcaserun-queued), [`finished`](#testcaserun-finished) [`started`](#testcaserun-started), [`finished`](#testcaserun-finished) | +| [`testSuiteRun`](#testsuiterun) | The execution of a software testSuite | [`queued`](#testsuiterun-queued), [`started`](#testsuiterun-started), [`finished`](#testsuiterun-finished) | +| [`testOutput`](#testoutput) | An output artifact produced by a test execution | [`published`](#testoutput-published) | -### `testCase` +### `testCaseRun` -A `testCase` is a process that performs a test against an input software artifact of some kind, for instance source code, a binary, a container image or else. -A `testCase` is the smallest unit of testing that the user wants to track. `testCases` are executed, and `testSuites` are for grouping purposes. +A `testCaseRun` is a process that performs a test against an input software artifact of some kind, for instance source code, a binary, a container image or else. +A `testCaseRun` is the smallest unit of testing that the user wants to track, `testSuiteRuns` are for grouping purposes. -| Field | Type | Description | Examples | -|--------|-----------------|-------------------------------------------------------------|--------------------------------------------------------------| -| id | `String` | Uniquely identifies the subject within the source. | `integration-test-abc`, `e2e-test1`, `scan-image1` | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | `staging/tekton`, `tekton-dev-123` | -| type | `String` | An optional type of test | `functional`, `unit`, `performance`, `security` | -| url | `URI-Reference` | An optional reference to view/access the specified testCase | `https://testkube.mycluster.internal/testCases/untitest-abc` | +| Field | Type | Description | Examples | +|----------------|---------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------|----------------------------------------------------| +| id | `String` | Uniquely identifies the subject within the source. | `integration-test-abc`, `e2e-test1`, `scan-image1` | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | `staging/tekton`, `tekton-dev-123` | +| type | `String` | An optional type of test | `functional`, `unit`, `performance`, `security` | +| environment | `Object` [`environment`](continuous-deployment-pipeline-events.md/#environment) | The environment in which this testCaseRun is executing | `dev`, `prod` | +| testCaseId | `String` | An optional testCase ID to enable collation of events related to a specific testCase definition | | | +| testSuiteRunId | `String` | An optional testSuiteRun ID to associate this testCaseRun with a containing testSuiteRun | | | -### `testSuite` +### `testSuiteRun` -A `testSuite` represents a set of one or more `testCases`. +A `testSuiteRun` represents the execution of a set of one or more `testCaseRuns`. -| Field | Type | Description | Examples | -|--------|-----------------|--------------------------------------------------------------|---------------------------------------------------------------| -| id | `String` | Uniquely identifies the subject within the source. | `my-testsuite`, `regression-123` | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | `staging/tekton`, `tekton-dev-123` | -| url | `URI-Reference` | An optional reference to view/access the specified testSuite | `https://testkube.mycluster.internal/testSuites/my-testsuite` | +| Field | Type | Description | Examples | +|-------------|---------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------|------------------------------------| +| id | `String` | Uniquely identifies the subject within the source. | `my-testsuite`, `regression-123` | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | `staging/tekton`, `tekton-dev-123` | +| environment | `Object` [`environment`](continuous-deployment-pipeline-events.md/#environment) | The environment in which this testSuiteRun is executing | `dev`, `prod` | +| testSuiteId | `String` | An optional testSuite ID to enable collation of events related to a specific testSuite definition | | | ### `testOutput` One or more `testOutput` artifacts are usually produced as the result of a test execution. -| Field | Type | Description | Examples | -|------------|-----------------|-------------------------------------------------------------------------|----------------------------------------------------------| -| id | `String` | Uniquely identifies the subject within the source. | `23123123` | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | `staging/tekton`, `tekton-dev-123` | -| type | `String` | The type of output, one of `report`, `video`, `image`, `log`, `other` | `video` | -| format | `String` | The Content-Type of the output artifact | `application/pdf`, `image/png`, `application/json` | -| uri | `URI-Reference` | A reference to retrieve the specified output artifact | `https://testkube.mycluster.internal/artifacts/23123123` | +| Field | Type | Description | Examples | +|---------------|-----------------|-----------------------------------------------------------------------------------|----------------------------------------------------------| +| id | `String` | Uniquely identifies the subject within the source. | `23123123` | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | `staging/tekton`, `tekton-dev-123` | +| type | `String` | The type of output, one of `report`, `video`, `image`, `log`, `other` | `video` | +| format | `String` | The Content-Type of the output artifact | `application/pdf`, `image/png`, `application/json` | +| uri | `URI-Reference` | A reference to retrieve the specified output artifact | `https://testkube.mycluster.internal/artifacts/23123123` | +| testCaseRunId | `String` | An optional testCaseRun ID to link this artifact to a specific testCase execution | | | + ## Events -### `testCase started` +### `testCaseRun queued` + +This event represents when a testCaseRun has been queued for execution - and is waiting for applicable preconditions +(resource availability, other tasks, etc.) to be met before actually executing. + +- Event Type: __`dev.cdevents.testcaserun.queued.0.1.0-draft`__ +- Predicate: queued +- Subject: [`testCaseRun`](#testcaserun) + +| Field | Type | Description | Examples | Required | +|---------|------------------------------|-----------------------------------------------------|-------------------------------------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| trigger | `Object` [trigger](#trigger) | What triggered this testCaseRun || + +### `testCaseRun started` This event represents a started testCase execution. -- Event Type: __`dev.cdevents.testcase.started.0.2.0-draft`__ +- Event Type: __`dev.cdevents.testcaserun.started.0.1.0-draft`__ - Predicate: started -- Subject: [`testCase`](#testcase) +- Subject: [`testCaseRun`](#testcaserun) -| Field | Type | Description | Examples | Required | -|--------------|-----------------|--------------------------------------------------------------------------------------------------------------|-------------------------------------------|----------| -| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | -| trigger.type | `String` | An optional cause of what triggered this testCase, one of `manual`, `pipeline`, `event`, `schedule`, `other` || -| trigger.uri | `URI-Reference` | An optional uri reference to what triggered this testCase || -| environment | `String` | An optional identifier for the environment in which this testCase is executing | `dev`, `prod` | -| executionId | `String` | An optional execution ID to enable handling of multiple simultaneous executions of this testCase | | | +| Field | Type | Description | Examples | Required | +|---------|------------------------------|-----------------------------------------------------|-------------------------------------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| trigger | `Object` [trigger](#trigger) | What triggered this testCaseRun || -### `testCase finished` +### `testCaseRun finished` This event represents a finished testCase execution. The event will contain the finished status and additional metadata as applicable. -- Event Type: __`dev.cdevents.testcase.finished.0.2.0-draft`__ +- Event Type: __`dev.cdevents.testcaserun.finished.0.1.0-draft`__ - Predicate: finished -- Subject: [`testCase`](#testcase) +- Subject: [`testCaseRun`](#testcaserun) + +| Field | Type | Description | Examples | Required | +|----------|-----------------|-------------------------------------------------------------------------------------|-------------------------------------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| status | `String` | The status of the testSuite execution, one of `pass`, `fail`, `abort`, `error` | | ✅ | +| severity | `String` | An optional severity if the test failed, one of `low`, `medium`, `high`, `critical` | `critical`, `low`, `medium`, `high` | +| reason | `String` | An optional reason related to the status of the execution | `Cancelled by user`, `Failed assertion` | | -| Field | Type | Description | Examples | Required | -|-------------|-----------------|--------------------------------------------------------------------------------------------------|-------------------------------------------|----------| -| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | -| status | `String` | The status of the testSuite execution, one of `passed`, `failed`, `aborted` | | ✅ | -| severity | `String` | An optional severity if the test failed, one of `critical`, `low`, `medium`, `high` | `critical`, `low`, `medium`, `high` | -| reason | `String` | An optional reason related to the status of the execution | `Cancelled by user`, `Failed assertion` | | -| executionId | `String` | An optional execution ID to enable handling of multiple simultaneous executions of this testCase | | | +### `testSuiteRun queued` +This event represents when a testSuiteRun has been queued for execution - and is waiting for applicable preconditions +(resource availability, other tasks, etc.) to be met before actually executing. -### `testSuite started` +- Event Type: __`dev.cdevents.testsuiterun.queued.0.1.0-draft`__ +- Predicate: queued +- Subject: [`testSuiteRun`](#testsuiterun) + +| Field | Type | Description | Examples | Required | +|---------|------------------------------|-----------------------------------------------------|-------------------------------------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| trigger | `Object` [trigger](#trigger) | What triggered this testSuiteRun || + +### `testSuiteRun started` This event represents a started testSuite execution. -- Event Type: __`dev.cdevents.testsuite.started.0.2.0-draft`__ +- Event Type: __`dev.cdevents.testsuiterun.started.0.1.0-draft`__ - Predicate: started -- Subject: [`testSuite`](#testsuite) +- Subject: [`testSuiteRun`](#testsuiterun) -| Field | Type | Description | Examples | Required | -|--------------|-----------------|---------------------------------------------------------------------------------------------------------------|---------------------------|----------| -| id | `String` | Uniquely identifies the subject within the source. | `unit`, `e2e`, `security` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | -| executionId | `String` | An optional execution ID to enable handling of multiple simultaneous executions | | | -| trigger.type | `String` | An optional cause of what triggered this testSuite, one of `manual`, `pipeline`, `event`, `schedule`, `other` ||| -| trigger.uri | `URI-Reference` | An optional uri reference to what triggered this testSuite ||| -| environment | `String` | An optional identifier for the environment in which this testSuite is executing | `dev`, `prod` | +| Field | Type | Description | Examples | Required | +|---------|------------------------------|-----------------------------------------------------|---------------------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `unit`, `e2e`, `security` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| trigger | `Object` [trigger](#trigger) | What triggered this testSuiteRun || -### `testSuite finished` +### `testSuiteRun finished` This event represents a finished testSuite execution. The event will contain the execution status and additional metadata as applicable. -- Event Type: __`dev.cdevents.testsuite.finished.0.1.0`__ +- Event Type: __`dev.cdevents.testsuiterun.finished.0.1.0-draft`__ - Predicate: finished -- Subject: [`testSuite`](#testsuite) +- Subject: [`testSuiteRun`](#testsuiterun) -| Field | Type | Description | Examples | Required | -|-------------|-----------------|---------------------------------------------------------------------------------------------------|----------------------------------------|----------| -| id | `String` | Uniquely identifies the subject within the source. | `unit`, `e2e`, `security` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | -| status | `String` | The status of the testSuite execution, one of `passed`, `failed`, `aborted` | | ✅ | -| severity | `String` | An optional severity if the testSuite failed, one of `critical`, `low`, `medium`, `high` | `critical`, `low`, `medium`, `high` | -| reason | `String` | An optional reason related to the status of the execution | `Cancelled by user`, `Failed testCase` | | -| executionId | `String` | An optional execution ID to enable handling of multiple simultaneous executions of this testSuite | | | +| Field | Type | Description | Examples | Required | +|----------|-----------------|-------------------------------------------------------------------------------------|----------------------------------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `unit`, `e2e`, `security` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| status | `String` | The status of the testSuite execution, one of `pass`, `fail`, `abort`, `error` | | ✅ | +| severity | `String` | An optional severity if the test failed, one of `low`, `medium`, `high`, `critical` | `critical`, `low`, `medium`, `high` | +| reason | `String` | An optional reason related to the status of the execution | `Cancelled by user`, `Failed testCase` | | ### `testOutput published` The event represents a test execution output artifact that has been published. -- Event Type: __`dev.cdevents.testoutput.published.0.2.0-draft`__ +- Event Type: __`dev.cdevents.testoutput.published.0.1.0-draft`__ - Predicate: published - Subject: [`testOutput`](#testoutput) -| Field | Type | Description | Examples | Required | -|-------------|-----------------|--------------------------------------------------------------------------------------------------|------------|----------| -| id | `String` | Uniquely identifies the subject within the source. | `12312334` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | -| executionId | `String` | An optional execution ID to link this artifact to a specific `testCase` or `testSuite` execution | | | +| Field | Type | Description | Examples | Required | +|---------------|-----------------|-----------------------------------------------------------------------------------|------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `12312334` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | ✅ | + +## Common Objects + +### `trigger` + +A `trigger` in this context is what started a corresponding testCaseRun/testSuiteRun. + +| Field | Type | Description | Examples | Required | +|-------------|-----------------|--------------------------------------------------------------------------------|----------|----------| +| type | `String` | The type of trigger, one of `manual`, `pipeline`, `event`, `schedule`, `other` | | ✅ | +| uri | `URI-Reference` | An optional uri reference to this trigger | | | + From af33f3d22296b02273e9bf01124a184b0d4baefd Mon Sep 17 00:00:00 2001 From: olensmar Date: Tue, 31 Jan 2023 11:39:05 +0100 Subject: [PATCH 14/37] minor fixes.. --- schemas/testoutputpublished.json | 2 +- testing-events.md | 44 ++++++++++++++++---------------- 2 files changed, 23 insertions(+), 23 deletions(-) diff --git a/schemas/testoutputpublished.json b/schemas/testoutputpublished.json index c3a0877b..bb1d669f 100644 --- a/schemas/testoutputpublished.json +++ b/schemas/testoutputpublished.json @@ -73,7 +73,7 @@ "format": "uri", "description": "an optional URI to the actual testOutput artifact" }, - "executionId": { + "testCaseRunId": { "type": "string", "description": "an optional identifier of the testCase or testSuite execution that produced this output" } diff --git a/testing-events.md b/testing-events.md index 24572601..05193708 100644 --- a/testing-events.md +++ b/testing-events.md @@ -18,11 +18,11 @@ Testing events covers the subjects and predicates related to test-execution perf This specification defines three subjects in this stage: `testCase`, `testSuite` and `testArtifact`. -| Subject | Description | Predicates | -|---------------------------------|-------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------| -| [`testCaseRun`](#testcaserun) | The execution of a software testCase | [`queued`](#testcaserun-queued), [`finished`](#testcaserun-finished) [`started`](#testcaserun-started), [`finished`](#testcaserun-finished) | -| [`testSuiteRun`](#testsuiterun) | The execution of a software testSuite | [`queued`](#testsuiterun-queued), [`started`](#testsuiterun-started), [`finished`](#testsuiterun-finished) | -| [`testOutput`](#testoutput) | An output artifact produced by a test execution | [`published`](#testoutput-published) | +| Subject | Description | Predicates | +|---------------------------------|----------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------| +| [`testCaseRun`](#testcaserun) | The execution of a software testCase | [`queued`](#testcaserun-queued), [`finished`](#testcaserun-finished) [`started`](#testcaserun-started), [`finished`](#testcaserun-finished) | +| [`testSuiteRun`](#testsuiterun) | The execution of a software testSuite | [`queued`](#testsuiterun-queued), [`started`](#testsuiterun-started), [`finished`](#testsuiterun-finished) | +| [`testOutput`](#testoutput) | An output artifact produced by a testCaseRun | [`published`](#testoutput-published) | ### `testCaseRun` @@ -40,7 +40,7 @@ A `testCaseRun` is the smallest unit of testing that the user wants to track, `t ### `testSuiteRun` -A `testSuiteRun` represents the execution of a set of one or more `testCaseRuns`. +A `testSuiteRun` represents the execution of a grouped set of one or more `testCaseRuns`. | Field | Type | Description | Examples | |-------------|---------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------|------------------------------------| @@ -53,14 +53,14 @@ A `testSuiteRun` represents the execution of a set of one or more `testCaseRuns` One or more `testOutput` artifacts are usually produced as the result of a test execution. -| Field | Type | Description | Examples | -|---------------|-----------------|-----------------------------------------------------------------------------------|----------------------------------------------------------| -| id | `String` | Uniquely identifies the subject within the source. | `23123123` | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | `staging/tekton`, `tekton-dev-123` | -| type | `String` | The type of output, one of `report`, `video`, `image`, `log`, `other` | `video` | -| format | `String` | The Content-Type of the output artifact | `application/pdf`, `image/png`, `application/json` | -| uri | `URI-Reference` | A reference to retrieve the specified output artifact | `https://testkube.mycluster.internal/artifacts/23123123` | -| testCaseRunId | `String` | An optional testCaseRun ID to link this artifact to a specific testCase execution | | | +| Field | Type | Description | Examples | +|---------------|-----------------|------------------------------------------------------------------------------|----------------------------------------------------------| +| id | `String` | Uniquely identifies the subject within the source. | `23123123` | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | `staging/tekton`, `tekton-dev-123` | +| type | `String` | The type of output, one of `report`, `video`, `image`, `log`, `other` | `video` | +| format | `String` | The Content-Type of the output artifact | `application/pdf`, `image/png`, `application/json` | +| uri | `URI-Reference` | A reference to retrieve the specified output artifact | `https://testkube.mycluster.internal/artifacts/23123123` | +| testCaseRunId | `String` | An optional testCaseRun ID to link this artifact to a specific `testCaseRun` | | | ## Events @@ -68,7 +68,7 @@ One or more `testOutput` artifacts are usually produced as the result of a test ### `testCaseRun queued` This event represents when a testCaseRun has been queued for execution - and is waiting for applicable preconditions -(resource availability, other tasks, etc.) to be met before actually executing. +(resource availability, other tasks, etc.) to be fulfilled before actually executing. - Event Type: __`dev.cdevents.testcaserun.queued.0.1.0-draft`__ - Predicate: queued @@ -102,13 +102,13 @@ This event represents a finished testCase execution. The event will contain the - Predicate: finished - Subject: [`testCaseRun`](#testcaserun) -| Field | Type | Description | Examples | Required | -|----------|-----------------|-------------------------------------------------------------------------------------|-------------------------------------------|----------| -| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | -| status | `String` | The status of the testSuite execution, one of `pass`, `fail`, `abort`, `error` | | ✅ | -| severity | `String` | An optional severity if the test failed, one of `low`, `medium`, `high`, `critical` | `critical`, `low`, `medium`, `high` | -| reason | `String` | An optional reason related to the status of the execution | `Cancelled by user`, `Failed assertion` | | +| Field | Type | Description | Examples | Required | +|----------|-----------------|-------------------------------------------------------------------------------------|------------------------------------------------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| status | `String` | The status of the testSuite execution, one of `pass`, `fail`, `abort`, `error` | | ✅ | +| severity | `String` | An optional severity if the test failed, one of `low`, `medium`, `high`, `critical` | `critical`, `low`, `medium`, `high` | +| reason | `String` | An optional reason related to the `status` of the execution | `Cancelled by user`, `Failed assertion`, `Timed out` | | ### `testSuiteRun queued` From 73cc269a18257ed7c70ea99c18612f2527a18d9c Mon Sep 17 00:00:00 2001 From: olensmar Date: Thu, 9 Feb 2023 12:24:39 +0100 Subject: [PATCH 15/37] fixed review comments: - added applicable subject properties to predicate property tables - changes testSuiteRunId / testCaseRunId to testSuiteRun / testCaseRun references - wording improvements.. --- README.md | 2 +- schemas/testcaserunfinished.json | 24 ++++-- schemas/testcaserunqueued.json | 26 ++++-- schemas/testcaserunstarted.json | 26 ++++-- schemas/testoutputpublished.json | 21 ++++- schemas/testsuiterunfinished.json | 35 ++++++-- schemas/testsuiterunqueued.json | 14 ++-- schemas/testsuiterunstarted.json | 14 ++-- testing-events.md | 131 +++++++++++++++++------------- 9 files changed, 197 insertions(+), 96 deletions(-) diff --git a/README.md b/README.md index 30177f5b..64d6f019 100644 --- a/README.md +++ b/README.md @@ -84,7 +84,7 @@ Handling Events associated with the health of the services deployed and running ### [Testing Events](./testing-events.md) -Handling Events associated with Test execution performed independently of CI/CD pipelines. +Handling Events associated with Test execution performed independently or as part of CI/CD pipelines. ### [CloudEvents Binding and Transport](./cloudevents-binding.md) diff --git a/schemas/testcaserunfinished.json b/schemas/testcaserunfinished.json index c0cffe3f..1b46382c 100644 --- a/schemas/testcaserunfinished.json +++ b/schemas/testcaserunfinished.json @@ -99,19 +99,33 @@ "id" ] }, - "testSuiteRunId": { - "type": "string", - "description": "an optional identifier of the containing testSuiteRun" + "testSuiteRun": { + "type": "object", + "description": "an optional reference to the containing testSuiteRun", + "properties": { + "id": { + "type": "string", + "minLength": 1 + }, + "source": { + "type": "string" + } + }, + "additionalProperties": false, + "required": [ + "id" + ] }, "testCaseId": { "type": "string", - "description": "an optional identifier of the queued testCase" + "description": "an optional external identifier of the queued testCase" } }, "additionalProperties": false, "type": "object", "required": [ - "status" + "status", + "environment" ] } }, diff --git a/schemas/testcaserunqueued.json b/schemas/testcaserunqueued.json index 38268c0a..96e4f6e5 100644 --- a/schemas/testcaserunqueued.json +++ b/schemas/testcaserunqueued.json @@ -115,17 +115,33 @@ "id" ] }, - "testSuiteRunId": { - "type": "string", - "description": "an optional identifier of the containing testSuiteRun" + "testSuiteRun": { + "description": "an optional reference to the containing testSuiteRun", + "properties": { + "id": { + "type": "string", + "minLength": 1 + }, + "source": { + "type": "string" + } + }, + "additionalProperties": false, + "type": "object", + "required": [ + "id" + ] }, "testCaseId": { "type": "string", - "description": "an optional identifier of the queued testCase" + "description": "an optional external identifier of the queued testCase" } }, "additionalProperties": false, - "type": "object" + "type": "object", + "required": [ + "environment" + ] } }, "additionalProperties": false, diff --git a/schemas/testcaserunstarted.json b/schemas/testcaserunstarted.json index d263684e..854be0ea 100644 --- a/schemas/testcaserunstarted.json +++ b/schemas/testcaserunstarted.json @@ -115,17 +115,33 @@ "id" ] }, - "testSuiteRunId": { - "type": "string", - "description": "an optional identifier of the containing testSuiteRun" + "testSuiteRun": { + "type": "object", + "description": "an optional reference to the containing testSuiteRun", + "properties": { + "id": { + "type": "string", + "minLength": 1 + }, + "source": { + "type": "string" + } + }, + "additionalProperties": false, + "required": [ + "id" + ] }, "testCaseId": { "type": "string", - "description": "an optional identifier of the running testCase" + "description": "an optional external identifier of the running testCase" } }, "additionalProperties": false, - "type": "object" + "type": "object", + "required": [ + "environment" + ] } }, "additionalProperties": false, diff --git a/schemas/testoutputpublished.json b/schemas/testoutputpublished.json index bb1d669f..deb1b1d4 100644 --- a/schemas/testoutputpublished.json +++ b/schemas/testoutputpublished.json @@ -71,11 +71,24 @@ "uri": { "type": "string", "format": "uri", - "description": "an optional URI to the actual testOutput artifact" + "description": "an optional URI reference to the actual testOutput artifact" }, - "testCaseRunId": { - "type": "string", - "description": "an optional identifier of the testCase or testSuite execution that produced this output" + "testCaseRun": { + "type": "object", + "description": "an optional reference to the testCaseRun that generated this output", + "properties": { + "id": { + "type": "string", + "minLength": 1 + }, + "source": { + "type": "string" + } + }, + "additionalProperties": false, + "required": [ + "id" + ] } }, "additionalProperties": false, diff --git a/schemas/testsuiterunfinished.json b/schemas/testsuiterunfinished.json index 62ff9ad1..657ea1d3 100644 --- a/schemas/testsuiterunfinished.json +++ b/schemas/testsuiterunfinished.json @@ -54,6 +54,34 @@ "content": { "properties": { "properties": { + "environment": { + "description": "the environment in which this testSuiteRun started", + "properties": { + "id": { + "type": "string", + "minLength": 1 + }, + "source": { + "type": "string" + }, + "name": { + "type": "string" + }, + "url": { + "type": "string", + "format": "uri" + } + }, + "additionalProperties": false, + "type": "object", + "required": [ + "id" + ] + }, + "testSuiteId": { + "type": "string", + "description": "an optional external identifier of the testSuite execution" + }, "status": { "type": "string", "enum": [ @@ -75,16 +103,13 @@ "reason": { "type": "string", "description": "an optional reason to why the testSuite ended as it did" - }, - "executionId": { - "type": "string", - "description": "an optional identifier of the testSuite execution" } }, "additionalProperties": false, "type": "object", "required": [ - "status" + "status", + "environment" ] }, "additionalProperties": false, diff --git a/schemas/testsuiterunqueued.json b/schemas/testsuiterunqueued.json index 6299406d..f8c7b684 100644 --- a/schemas/testsuiterunqueued.json +++ b/schemas/testsuiterunqueued.json @@ -53,11 +53,6 @@ }, "content": { "properties": { - "uri": { - "type": "string", - "format": "uri", - "description": "an optional URI to the actual testSuite" - }, "trigger": { "type": "object", "description": "an optional definition of what triggered this testCase execution", @@ -102,13 +97,16 @@ "id" ] }, - "executionId": { + "testSuiteId": { "type": "string", - "description": "an optional identifier of the testSuite execution" + "description": "an optional external identifier of the testSuite execution" } }, "additionalProperties": false, - "type": "object" + "type": "object", + "required": [ + "environment" + ] } }, "additionalProperties": false, diff --git a/schemas/testsuiterunstarted.json b/schemas/testsuiterunstarted.json index 3d42c46d..9601fb02 100644 --- a/schemas/testsuiterunstarted.json +++ b/schemas/testsuiterunstarted.json @@ -53,11 +53,6 @@ }, "content": { "properties": { - "uri": { - "type": "string", - "format": "uri", - "description": "an optional URI to the actual testSuite" - }, "trigger": { "type": "object", "description": "an optional definition of what triggered this testCase execution", @@ -102,13 +97,16 @@ "id" ] }, - "executionId": { + "testSuiteId": { "type": "string", - "description": "an optional identifier of the testSuite execution" + "description": "an optional external identifier of the testSuite execution" } }, "additionalProperties": false, - "type": "object" + "type": "object", + "required": [ + "environment" + ] } }, "additionalProperties": false, diff --git a/testing-events.md b/testing-events.md index 05193708..d40ba599 100644 --- a/testing-events.md +++ b/testing-events.md @@ -29,14 +29,14 @@ This specification defines three subjects in this stage: `testCase`, `testSuite` A `testCaseRun` is a process that performs a test against an input software artifact of some kind, for instance source code, a binary, a container image or else. A `testCaseRun` is the smallest unit of testing that the user wants to track, `testSuiteRuns` are for grouping purposes. -| Field | Type | Description | Examples | -|----------------|---------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------|----------------------------------------------------| -| id | `String` | Uniquely identifies the subject within the source. | `integration-test-abc`, `e2e-test1`, `scan-image1` | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | `staging/tekton`, `tekton-dev-123` | -| type | `String` | An optional type of test | `functional`, `unit`, `performance`, `security` | -| environment | `Object` [`environment`](continuous-deployment-pipeline-events.md/#environment) | The environment in which this testCaseRun is executing | `dev`, `prod` | -| testCaseId | `String` | An optional testCase ID to enable collation of events related to a specific testCase definition | | | -| testSuiteRunId | `String` | An optional testSuiteRun ID to associate this testCaseRun with a containing testSuiteRun | | | +| Field | Type | Description | Examples | +|--------------|---------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------|----------------------------------------------------| +| id | `String` | Uniquely identifies the subject within the source. | `integration-test-abc`, `e2e-test1`, `scan-image1` | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | `staging/tekton`, `tekton-dev-123` | +| type | `String` | An optional type of test | `functional`, `unit`, `performance`, `security` | +| environment | `Object` [`environment`](continuous-deployment-pipeline-events.md/#environment) | The environment in which this testCaseRun is executing | `dev`, `prod` | +| testCaseId | `String` | An optional testCase ID to enable collation of events related to a specific testCase definition | | | +| testSuiteRun | `Object` [`testSuiteRun`](#testsuiterun) | An optional testSuiteRun to associate this testCaseRun with a containing testSuiteRun | | | ### `testSuiteRun` @@ -53,15 +53,14 @@ A `testSuiteRun` represents the execution of a grouped set of one or more `testC One or more `testOutput` artifacts are usually produced as the result of a test execution. -| Field | Type | Description | Examples | -|---------------|-----------------|------------------------------------------------------------------------------|----------------------------------------------------------| -| id | `String` | Uniquely identifies the subject within the source. | `23123123` | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | `staging/tekton`, `tekton-dev-123` | -| type | `String` | The type of output, one of `report`, `video`, `image`, `log`, `other` | `video` | -| format | `String` | The Content-Type of the output artifact | `application/pdf`, `image/png`, `application/json` | -| uri | `URI-Reference` | A reference to retrieve the specified output artifact | `https://testkube.mycluster.internal/artifacts/23123123` | -| testCaseRunId | `String` | An optional testCaseRun ID to link this artifact to a specific `testCaseRun` | | | - +| Field | Type | Description | Examples | +|-------------|----------------------------------------|---------------------------------------------------------------------------|----------------------------------------------------------| +| id | `String` | Uniquely identifies the subject within the source. | `23123123` | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | `staging/tekton`, `tekton-dev-123` | +| type | `String` | The type of output, one of `report`, `video`, `image`, `log`, `other` | `video` | +| format | `String` | The Content-Type of the output artifact | `application/pdf`, `image/png`, `application/json` | +| uri | `URI-Reference` | A reference to retrieve the specified output artifact | `https://testkube.mycluster.internal/artifacts/23123123` | +| testCaseRun | `Object` [`testCaseRun`](#testcaserun) | An optional testCaseRun to link this artifact to a specific `testCaseRun` | | | ## Events @@ -74,11 +73,15 @@ This event represents when a testCaseRun has been queued for execution - and is - Predicate: queued - Subject: [`testCaseRun`](#testcaserun) -| Field | Type | Description | Examples | Required | -|---------|------------------------------|-----------------------------------------------------|-------------------------------------------|----------| -| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | -| trigger | `Object` [trigger](#trigger) | What triggered this testCaseRun || +| Field | Type | Description | Examples | Required | +|--------------|---------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------|-------------------------------------------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| type | `String` | An optional type of test | `functional`, `unit`, `performance`, `security` | +| environment | `Object` [`environment`](continuous-deployment-pipeline-events.md/#environment) | The environment in which this testCaseRun is queued | `dev`, `prod` | ✅ | +| testCaseId | `String` | An optional external identifier to enable collation of events related to a specific testCase definition | | | +| testSuiteRun | `Object` [`testSuiteRun`](#testsuiterun) | An optional testSuiteRun to associate this testCaseRun with a containing testSuiteRun | | | +| trigger | `Object` [trigger](#trigger) | What triggered the queuing of this testCaseRun || ### `testCaseRun started` @@ -88,11 +91,15 @@ This event represents a started testCase execution. - Predicate: started - Subject: [`testCaseRun`](#testcaserun) -| Field | Type | Description | Examples | Required | -|---------|------------------------------|-----------------------------------------------------|-------------------------------------------|----------| -| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | -| trigger | `Object` [trigger](#trigger) | What triggered this testCaseRun || +| Field | Type | Description | Examples | Required | +|--------------|---------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------|-------------------------------------------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| type | `String` | An optional type of test | `functional`, `unit`, `performance`, `security` | +| environment | `Object` [`environment`](continuous-deployment-pipeline-events.md/#environment) | The environment in which this testCaseRun is running | `dev`, `prod` | ✅ | +| testCaseId | `String` | An optional external identifier to enable collation of events related to a specific testCase definition | | | +| testSuiteRun | `Object` [`testSuiteRun`](#testsuiterun) | An optional testSuiteRun to associate this testCaseRun with a containing testSuiteRun | | | +| trigger | `Object` [trigger](#trigger) | What triggered this testCaseRun || ### `testCaseRun finished` @@ -102,13 +109,17 @@ This event represents a finished testCase execution. The event will contain the - Predicate: finished - Subject: [`testCaseRun`](#testcaserun) -| Field | Type | Description | Examples | Required | -|----------|-----------------|-------------------------------------------------------------------------------------|------------------------------------------------------|----------| -| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | -| status | `String` | The status of the testSuite execution, one of `pass`, `fail`, `abort`, `error` | | ✅ | -| severity | `String` | An optional severity if the test failed, one of `low`, `medium`, `high`, `critical` | `critical`, `low`, `medium`, `high` | -| reason | `String` | An optional reason related to the `status` of the execution | `Cancelled by user`, `Failed assertion`, `Timed out` | | +| Field | Type | Description | Examples | Required | +|--------------|---------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------|------------------------------------------------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| type | `String` | An optional type of test | `functional`, `unit`, `performance`, `security` | +| environment | `Object` [`environment`](continuous-deployment-pipeline-events.md/#environment) | The environment in which this testCaseRun was running | `dev`, `prod` | ✅ | +| testCaseId | `String` | An optional external identifier to enable collation of events related to a specific testCase definition | | | +| testSuiteRun | `Object` [`testSuiteRun`](#testsuiterun) | An optional testSuiteRun to associate this testCaseRun with a containing testSuiteRun | | | +| status | `String` | The status of the testSuite execution, one of `pass`, `fail`, `abort`, `error` | | ✅ | +| severity | `String` | An optional severity if the test failed, one of `low`, `medium`, `high`, `critical` | `critical`, `low`, `medium`, `high` | +| reason | `String` | An optional reason related to the `status` of the execution | `Cancelled by user`, `Failed assertion`, `Timed out` | | ### `testSuiteRun queued` @@ -119,11 +130,13 @@ This event represents when a testSuiteRun has been queued for execution - and is - Predicate: queued - Subject: [`testSuiteRun`](#testsuiterun) -| Field | Type | Description | Examples | Required | -|---------|------------------------------|-----------------------------------------------------|-------------------------------------------|----------| -| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | -| trigger | `Object` [trigger](#trigger) | What triggered this testSuiteRun || +| Field | Type | Description | Examples | Required | +|-------------|---------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------|-------------------------------------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| environment | `Object` [`environment`](continuous-deployment-pipeline-events.md/#environment) | The environment in which this testSuiteRun is queued | `dev`, `prod` | ✅ | +| testSuiteId | `String` | An optional external identifier to enable collation of events related to a specific testSuite definition | | | +| trigger | `Object` [trigger](#trigger) | What triggered this testSuiteRun || ### `testSuiteRun started` @@ -133,11 +146,13 @@ This event represents a started testSuite execution. - Predicate: started - Subject: [`testSuiteRun`](#testsuiterun) -| Field | Type | Description | Examples | Required | -|---------|------------------------------|-----------------------------------------------------|---------------------------|----------| -| id | `String` | Uniquely identifies the subject within the source. | `unit`, `e2e`, `security` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | -| trigger | `Object` [trigger](#trigger) | What triggered this testSuiteRun || +| Field | Type | Description | Examples | Required | +|-------------|---------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------|---------------------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `unit`, `e2e`, `security` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| environment | `Object` [`environment`](continuous-deployment-pipeline-events.md/#environment) | The environment in which this testSuiteRun is running | `dev`, `prod` | ✅ | +| testSuiteId | `String` | An optional external identifier to enable collation of events related to a specific testSuite definition | | | +| trigger | `Object` [trigger](#trigger) | What triggered this testSuiteRun || ### `testSuiteRun finished` @@ -147,13 +162,15 @@ This event represents a finished testSuite execution. The event will contain the - Predicate: finished - Subject: [`testSuiteRun`](#testsuiterun) -| Field | Type | Description | Examples | Required | -|----------|-----------------|-------------------------------------------------------------------------------------|----------------------------------------|----------| -| id | `String` | Uniquely identifies the subject within the source. | `unit`, `e2e`, `security` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | -| status | `String` | The status of the testSuite execution, one of `pass`, `fail`, `abort`, `error` | | ✅ | -| severity | `String` | An optional severity if the test failed, one of `low`, `medium`, `high`, `critical` | `critical`, `low`, `medium`, `high` | -| reason | `String` | An optional reason related to the status of the execution | `Cancelled by user`, `Failed testCase` | | +| Field | Type | Description | Examples | Required | +|-------------|---------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------|----------------------------------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `unit`, `e2e`, `security` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| environment | `Object` [`environment`](continuous-deployment-pipeline-events.md/#environment) | The environment in which this testSuiteRun was running | `dev`, `prod` | ✅ | +| testSuiteId | `String` | An optional testSuite ID to enable collation of events related to a specific testSuite definition | | | +| status | `String` | The status of the testSuite execution, one of `pass`, `fail`, `abort`, `error` | | ✅ | +| severity | `String` | An optional severity if the test failed, one of `low`, `medium`, `high`, `critical` | `critical`, `low`, `medium`, `high` | +| reason | `String` | An optional reason related to the status of the execution | `Cancelled by user`, `Failed testCase` | | ### `testOutput published` @@ -163,10 +180,14 @@ The event represents a test execution output artifact that has been published. - Predicate: published - Subject: [`testOutput`](#testoutput) -| Field | Type | Description | Examples | Required | -|---------------|-----------------|-----------------------------------------------------------------------------------|------------|----------| -| id | `String` | Uniquely identifies the subject within the source. | `12312334` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | ✅ | +| Field | Type | Description | Examples | Required | +|-------------|----------------------------------------|---------------------------------------------------------------------------|----------------------------------------------------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `12312334` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | ✅ | +| type | `String` | The type of output, one of `report`, `video`, `image`, `log`, `other` | `video` | ✅ | +| format | `String` | The Content-Type of the output artifact | `application/pdf`, `image/png`, `application/json` | ✅ | +| uri | `URI-Reference` | A reference to retrieve the specified output artifact | `https://testkube.mycluster.internal/artifacts/23123123` | ✅ | +| testCaseRun | `Object` [`testCaseRun`](#testcaserun) | An optional testCaseRun to link this artifact to a specific `testCaseRun` | | | ## Common Objects From b00eeed6cc7bebc129535c63202dd76dcd239a54 Mon Sep 17 00:00:00 2001 From: olensmar Date: Wed, 8 Mar 2023 11:27:04 +0100 Subject: [PATCH 16/37] review comment: changed abort to cancel --- schemas/testcaserunfinished.json | 2 +- schemas/testsuiterunfinished.json | 2 +- testing-events.md | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/schemas/testcaserunfinished.json b/schemas/testcaserunfinished.json index 1b46382c..b1d774f6 100644 --- a/schemas/testcaserunfinished.json +++ b/schemas/testcaserunfinished.json @@ -58,7 +58,7 @@ "enum": [ "pass", "fail", - "abort", + "cancel", "error" ] }, diff --git a/schemas/testsuiterunfinished.json b/schemas/testsuiterunfinished.json index 657ea1d3..04ae82a9 100644 --- a/schemas/testsuiterunfinished.json +++ b/schemas/testsuiterunfinished.json @@ -87,7 +87,7 @@ "enum": [ "pass", "fail", - "abort", + "cancel", "error" ] }, diff --git a/testing-events.md b/testing-events.md index d40ba599..a386a009 100644 --- a/testing-events.md +++ b/testing-events.md @@ -117,7 +117,7 @@ This event represents a finished testCase execution. The event will contain the | environment | `Object` [`environment`](continuous-deployment-pipeline-events.md/#environment) | The environment in which this testCaseRun was running | `dev`, `prod` | ✅ | | testCaseId | `String` | An optional external identifier to enable collation of events related to a specific testCase definition | | | | testSuiteRun | `Object` [`testSuiteRun`](#testsuiterun) | An optional testSuiteRun to associate this testCaseRun with a containing testSuiteRun | | | -| status | `String` | The status of the testSuite execution, one of `pass`, `fail`, `abort`, `error` | | ✅ | +| status | `String` | The status of the testSuite execution, one of `pass`, `fail`, `cancel`, `error` | | ✅ | | severity | `String` | An optional severity if the test failed, one of `low`, `medium`, `high`, `critical` | `critical`, `low`, `medium`, `high` | | reason | `String` | An optional reason related to the `status` of the execution | `Cancelled by user`, `Failed assertion`, `Timed out` | | @@ -168,7 +168,7 @@ This event represents a finished testSuite execution. The event will contain the | source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | | environment | `Object` [`environment`](continuous-deployment-pipeline-events.md/#environment) | The environment in which this testSuiteRun was running | `dev`, `prod` | ✅ | | testSuiteId | `String` | An optional testSuite ID to enable collation of events related to a specific testSuite definition | | | -| status | `String` | The status of the testSuite execution, one of `pass`, `fail`, `abort`, `error` | | ✅ | +| status | `String` | The status of the testSuite execution, one of `pass`, `fail`, `cancel`, `error` | | ✅ | | severity | `String` | An optional severity if the test failed, one of `low`, `medium`, `high`, `critical` | `critical`, `low`, `medium`, `high` | | reason | `String` | An optional reason related to the status of the execution | `Cancelled by user`, `Failed testCase` | | From 68328777613563fd942c6688eaad320f7d81d65e Mon Sep 17 00:00:00 2001 From: olensmar Date: Wed, 8 Mar 2023 11:40:42 +0100 Subject: [PATCH 17/37] review comments: fixed environment examples and spec errors --- testing-events.md | 136 +++++++++++++++++++++++----------------------- 1 file changed, 68 insertions(+), 68 deletions(-) diff --git a/testing-events.md b/testing-events.md index a386a009..9bce912f 100644 --- a/testing-events.md +++ b/testing-events.md @@ -16,38 +16,38 @@ Testing events covers the subjects and predicates related to test-execution perf ## Subjects -This specification defines three subjects in this stage: `testCase`, `testSuite` and `testArtifact`. +This specification defines three subjects in this stage: `testCase`, `testSuite` and `testOutput`. -| Subject | Description | Predicates | -|---------------------------------|----------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------| -| [`testCaseRun`](#testcaserun) | The execution of a software testCase | [`queued`](#testcaserun-queued), [`finished`](#testcaserun-finished) [`started`](#testcaserun-started), [`finished`](#testcaserun-finished) | -| [`testSuiteRun`](#testsuiterun) | The execution of a software testSuite | [`queued`](#testsuiterun-queued), [`started`](#testsuiterun-started), [`finished`](#testsuiterun-finished) | -| [`testOutput`](#testoutput) | An output artifact produced by a testCaseRun | [`published`](#testoutput-published) | +| Subject | Description | Predicates | +|---------------------------------|----------------------------------------------|------------------------------------------------------------------------------------------------------------| +| [`testCaseRun`](#testcaserun) | The execution of a software testCase | [`queued`](#testcaserun-queued), [`started`](#testcaserun-started) [`finished`](#testcaserun-finished) | +| [`testSuiteRun`](#testsuiterun) | The execution of a software testSuite | [`queued`](#testsuiterun-queued), [`started`](#testsuiterun-started), [`finished`](#testsuiterun-finished) | +| [`testOutput`](#testoutput) | An output artifact produced by a testCaseRun | [`published`](#testoutput-published) | ### `testCaseRun` A `testCaseRun` is a process that performs a test against an input software artifact of some kind, for instance source code, a binary, a container image or else. A `testCaseRun` is the smallest unit of testing that the user wants to track, `testSuiteRuns` are for grouping purposes. -| Field | Type | Description | Examples | -|--------------|---------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------|----------------------------------------------------| -| id | `String` | Uniquely identifies the subject within the source. | `integration-test-abc`, `e2e-test1`, `scan-image1` | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | `staging/tekton`, `tekton-dev-123` | -| type | `String` | An optional type of test | `functional`, `unit`, `performance`, `security` | -| environment | `Object` [`environment`](continuous-deployment-pipeline-events.md/#environment) | The environment in which this testCaseRun is executing | `dev`, `prod` | -| testCaseId | `String` | An optional testCase ID to enable collation of events related to a specific testCase definition | | | -| testSuiteRun | `Object` [`testSuiteRun`](#testsuiterun) | An optional testSuiteRun to associate this testCaseRun with a containing testSuiteRun | | | +| Field | Type | Description | Examples | +|--------------|---------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------|---------------------------------------------------------------| +| id | `String` | Uniquely identifies the subject within the source. | `integration-test-abc`, `e2e-test1`, `scan-image1` | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | `staging/tekton`, `tekton-dev-123` | +| type | `String` | An optional type of test | `functional`, `unit`, `performance`, `security` | +| environment | `Object` [`environment`](continuous-deployment-pipeline-events.md/#environment) | The environment in which this testCaseRun is executing | `{"id": "1234"}`, `{"id": "dev", "source": "tekton-dev-123"}` | +| testCaseId | `String` | An optional testCase ID to enable collation of events related to a specific testCase definition | | | +| testSuiteRun | `Object` [`testSuiteRun`](#testsuiterun) | An optional testSuiteRun to associate this testCaseRun with a containing testSuiteRun | | | ### `testSuiteRun` A `testSuiteRun` represents the execution of a grouped set of one or more `testCaseRuns`. -| Field | Type | Description | Examples | -|-------------|---------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------|------------------------------------| -| id | `String` | Uniquely identifies the subject within the source. | `my-testsuite`, `regression-123` | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | `staging/tekton`, `tekton-dev-123` | -| environment | `Object` [`environment`](continuous-deployment-pipeline-events.md/#environment) | The environment in which this testSuiteRun is executing | `dev`, `prod` | -| testSuiteId | `String` | An optional testSuite ID to enable collation of events related to a specific testSuite definition | | | +| Field | Type | Description | Examples | +|-------------|---------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------|---------------------------------------------------------------| +| id | `String` | Uniquely identifies the subject within the source. | `my-testsuite`, `regression-123` | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | `staging/tekton`, `tekton-dev-123` | +| environment | `Object` [`environment`](continuous-deployment-pipeline-events.md/#environment) | The environment in which this testSuiteRun is executing | `{"id": "1234"}`, `{"id": "dev", "source": "tekton-dev-123"}` | +| testSuiteId | `String` | An optional testSuite ID to enable collation of events related to a specific testSuite definition | | | ### `testOutput` @@ -73,14 +73,14 @@ This event represents when a testCaseRun has been queued for execution - and is - Predicate: queued - Subject: [`testCaseRun`](#testcaserun) -| Field | Type | Description | Examples | Required | -|--------------|---------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------|-------------------------------------------------|----------| -| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | -| type | `String` | An optional type of test | `functional`, `unit`, `performance`, `security` | -| environment | `Object` [`environment`](continuous-deployment-pipeline-events.md/#environment) | The environment in which this testCaseRun is queued | `dev`, `prod` | ✅ | -| testCaseId | `String` | An optional external identifier to enable collation of events related to a specific testCase definition | | | -| testSuiteRun | `Object` [`testSuiteRun`](#testsuiterun) | An optional testSuiteRun to associate this testCaseRun with a containing testSuiteRun | | | +| Field | Type | Description | Examples | Required | +|--------------|---------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------|---------------------------------------------------------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| type | `String` | An optional type of test | `functional`, `unit`, `performance`, `security` | +| environment | `Object` [`environment`](continuous-deployment-pipeline-events.md/#environment) | The environment in which this testCaseRun is queued | `{"id": "1234"}`, `{"id": "dev", "source": "tekton-dev-123"}` | ✅ | +| testCaseId | `String` | An optional external identifier to enable collation of events related to a specific testCase definition | | | +| testSuiteRun | `Object` [`testSuiteRun`](#testsuiterun) | An optional testSuiteRun to associate this testCaseRun with a containing testSuiteRun | | | | trigger | `Object` [trigger](#trigger) | What triggered the queuing of this testCaseRun || ### `testCaseRun started` @@ -91,14 +91,14 @@ This event represents a started testCase execution. - Predicate: started - Subject: [`testCaseRun`](#testcaserun) -| Field | Type | Description | Examples | Required | -|--------------|---------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------|-------------------------------------------------|----------| -| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | -| type | `String` | An optional type of test | `functional`, `unit`, `performance`, `security` | -| environment | `Object` [`environment`](continuous-deployment-pipeline-events.md/#environment) | The environment in which this testCaseRun is running | `dev`, `prod` | ✅ | -| testCaseId | `String` | An optional external identifier to enable collation of events related to a specific testCase definition | | | -| testSuiteRun | `Object` [`testSuiteRun`](#testsuiterun) | An optional testSuiteRun to associate this testCaseRun with a containing testSuiteRun | | | +| Field | Type | Description | Examples | Required | +|--------------|---------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------|---------------------------------------------------------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| type | `String` | An optional type of test | `functional`, `unit`, `performance`, `security` | +| environment | `Object` [`environment`](continuous-deployment-pipeline-events.md/#environment) | The environment in which this testCaseRun is running | `{"id": "1234"}`, `{"id": "dev", "source": "tekton-dev-123"}` | ✅ | +| testCaseId | `String` | An optional external identifier to enable collation of events related to a specific testCase definition | | | +| testSuiteRun | `Object` [`testSuiteRun`](#testsuiterun) | An optional testSuiteRun to associate this testCaseRun with a containing testSuiteRun | | | | trigger | `Object` [trigger](#trigger) | What triggered this testCaseRun || ### `testCaseRun finished` @@ -109,17 +109,17 @@ This event represents a finished testCase execution. The event will contain the - Predicate: finished - Subject: [`testCaseRun`](#testcaserun) -| Field | Type | Description | Examples | Required | -|--------------|---------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------|------------------------------------------------------|----------| -| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | -| type | `String` | An optional type of test | `functional`, `unit`, `performance`, `security` | -| environment | `Object` [`environment`](continuous-deployment-pipeline-events.md/#environment) | The environment in which this testCaseRun was running | `dev`, `prod` | ✅ | -| testCaseId | `String` | An optional external identifier to enable collation of events related to a specific testCase definition | | | -| testSuiteRun | `Object` [`testSuiteRun`](#testsuiterun) | An optional testSuiteRun to associate this testCaseRun with a containing testSuiteRun | | | -| status | `String` | The status of the testSuite execution, one of `pass`, `fail`, `cancel`, `error` | | ✅ | -| severity | `String` | An optional severity if the test failed, one of `low`, `medium`, `high`, `critical` | `critical`, `low`, `medium`, `high` | -| reason | `String` | An optional reason related to the `status` of the execution | `Cancelled by user`, `Failed assertion`, `Timed out` | | +| Field | Type | Description | Examples | Required | +|--------------|---------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------|---------------------------------------------------------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| type | `String` | An optional type of test | `functional`, `unit`, `performance`, `security` | +| environment | `Object` [`environment`](continuous-deployment-pipeline-events.md/#environment) | The environment in which this testCaseRun was running | `{"id": "1234"}`, `{"id": "dev", "source": "tekton-dev-123"}` | ✅ | +| testCaseId | `String` | An optional external identifier to enable collation of events related to a specific testCase definition | | | +| testSuiteRun | `Object` [`testSuiteRun`](#testsuiterun) | An optional testSuiteRun to associate this testCaseRun with a containing testSuiteRun | | | +| status | `String` | The status of the testSuite execution, one of `pass`, `fail`, `cancel`, `error` | | ✅ | +| severity | `String` | An optional severity if the test failed, one of `low`, `medium`, `high`, `critical` | `critical`, `low`, `medium`, `high` | +| reason | `String` | An optional reason related to the `status` of the execution | `Cancelled by user`, `Failed assertion`, `Timed out` | | ### `testSuiteRun queued` @@ -130,12 +130,12 @@ This event represents when a testSuiteRun has been queued for execution - and is - Predicate: queued - Subject: [`testSuiteRun`](#testsuiterun) -| Field | Type | Description | Examples | Required | -|-------------|---------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------|-------------------------------------------|----------| -| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | -| environment | `Object` [`environment`](continuous-deployment-pipeline-events.md/#environment) | The environment in which this testSuiteRun is queued | `dev`, `prod` | ✅ | -| testSuiteId | `String` | An optional external identifier to enable collation of events related to a specific testSuite definition | | | +| Field | Type | Description | Examples | Required | +|-------------|---------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------|---------------------------------------------------------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| environment | `Object` [`environment`](continuous-deployment-pipeline-events.md/#environment) | The environment in which this testSuiteRun is queued | `{"id": "1234"}`, `{"id": "dev", "source": "tekton-dev-123"}` | ✅ | +| testSuiteId | `String` | An optional external identifier to enable collation of events related to a specific testSuite definition | | | | trigger | `Object` [trigger](#trigger) | What triggered this testSuiteRun || ### `testSuiteRun started` @@ -146,12 +146,12 @@ This event represents a started testSuite execution. - Predicate: started - Subject: [`testSuiteRun`](#testsuiterun) -| Field | Type | Description | Examples | Required | -|-------------|---------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------|---------------------------|----------| -| id | `String` | Uniquely identifies the subject within the source. | `unit`, `e2e`, `security` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | -| environment | `Object` [`environment`](continuous-deployment-pipeline-events.md/#environment) | The environment in which this testSuiteRun is running | `dev`, `prod` | ✅ | -| testSuiteId | `String` | An optional external identifier to enable collation of events related to a specific testSuite definition | | | +| Field | Type | Description | Examples | Required | +|-------------|---------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------|---------------------------------------------------------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `unit`, `e2e`, `security` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| environment | `Object` [`environment`](continuous-deployment-pipeline-events.md/#environment) | The environment in which this testSuiteRun is running | `{"id": "1234"}`, `{"id": "dev", "source": "tekton-dev-123"}` | ✅ | +| testSuiteId | `String` | An optional external identifier to enable collation of events related to a specific testSuite definition | | | | trigger | `Object` [trigger](#trigger) | What triggered this testSuiteRun || ### `testSuiteRun finished` @@ -162,15 +162,15 @@ This event represents a finished testSuite execution. The event will contain the - Predicate: finished - Subject: [`testSuiteRun`](#testsuiterun) -| Field | Type | Description | Examples | Required | -|-------------|---------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------|----------------------------------------|----------| -| id | `String` | Uniquely identifies the subject within the source. | `unit`, `e2e`, `security` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | -| environment | `Object` [`environment`](continuous-deployment-pipeline-events.md/#environment) | The environment in which this testSuiteRun was running | `dev`, `prod` | ✅ | -| testSuiteId | `String` | An optional testSuite ID to enable collation of events related to a specific testSuite definition | | | -| status | `String` | The status of the testSuite execution, one of `pass`, `fail`, `cancel`, `error` | | ✅ | -| severity | `String` | An optional severity if the test failed, one of `low`, `medium`, `high`, `critical` | `critical`, `low`, `medium`, `high` | -| reason | `String` | An optional reason related to the status of the execution | `Cancelled by user`, `Failed testCase` | | +| Field | Type | Description | Examples | Required | +|-------------|---------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------|---------------------------------------------------------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `unit`, `e2e`, `security` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| environment | `Object` [`environment`](continuous-deployment-pipeline-events.md/#environment) | The environment in which this testSuiteRun was running | `{"id": "1234"}`, `{"id": "dev", "source": "tekton-dev-123"}` | ✅ | +| testSuiteId | `String` | An optional testSuite ID to enable collation of events related to a specific testSuite definition | | | +| status | `String` | The status of the testSuite execution, one of `pass`, `fail`, `cancel`, `error` | | ✅ | +| severity | `String` | An optional severity if the test failed, one of `low`, `medium`, `high`, `critical` | `critical`, `low`, `medium`, `high` | +| reason | `String` | An optional reason related to the status of the execution | `Cancelled by user`, `Failed testCase` | | ### `testOutput published` From f1e5b551d0a3368a39a846b411be5dce961b3849 Mon Sep 17 00:00:00 2001 From: olensmar Date: Wed, 29 Mar 2023 16:47:27 +0200 Subject: [PATCH 18/37] review comments: extracted objects for testCase/testSuite, added examples, corresponding schema fixes --- schemas/testcaserunfinished.json | 34 +++++- schemas/testcaserunqueued.json | 52 ++++---- schemas/testcaserunstarted.json | 58 +++++---- schemas/testsuiterunfinished.json | 21 +++- schemas/testsuiterunqueued.json | 21 +++- schemas/testsuiterunstarted.json | 21 +++- testing-events.md | 191 +++++++++++++++++------------- 7 files changed, 260 insertions(+), 138 deletions(-) diff --git a/schemas/testcaserunfinished.json b/schemas/testcaserunfinished.json index b1d774f6..71c410a5 100644 --- a/schemas/testcaserunfinished.json +++ b/schemas/testcaserunfinished.json @@ -116,9 +116,37 @@ "id" ] }, - "testCaseId": { - "type": "string", - "description": "an optional external identifier of the queued testCase" + "testCase": { + "description": "the testCase which this testCaseRun is executing", + "properties": { + "id": { + "type": "string", + "minLength": 1 + }, + "source": { + "type": "string" + }, + "name": { + "type": "string" + }, + "type": { + "type": "string", + "enum": [ + "performance", + "functional", + "unit", + "security", + "compliance", + "integration", + "e2e", + "other" + ] + }, + "url": { + "type": "string", + "format": "uri" + } + } } }, "additionalProperties": false, diff --git a/schemas/testcaserunqueued.json b/schemas/testcaserunqueued.json index 96e4f6e5..f836636f 100644 --- a/schemas/testcaserunqueued.json +++ b/schemas/testcaserunqueued.json @@ -53,24 +53,6 @@ }, "content": { "properties": { - "type": { - "type": "string", - "enum": [ - "performance", - "functional", - "unit", - "security", - "compliance", - "integration", - "e2e", - "other" - ] - }, - "uri": { - "type": "string", - "format": "uri", - "description": "an optional URI to the actual testCase" - }, "trigger": { "type": "object", "description": "an optional definition of what triggered this testCase queueing", @@ -132,9 +114,37 @@ "id" ] }, - "testCaseId": { - "type": "string", - "description": "an optional external identifier of the queued testCase" + "testCase": { + "description": "the testCase which this testCaseRun is executing", + "properties": { + "id": { + "type": "string", + "minLength": 1 + }, + "source": { + "type": "string" + }, + "name": { + "type": "string" + }, + "type": { + "type": "string", + "enum": [ + "performance", + "functional", + "unit", + "security", + "compliance", + "integration", + "e2e", + "other" + ] + }, + "url": { + "type": "string", + "format": "uri" + } + } } }, "additionalProperties": false, diff --git a/schemas/testcaserunstarted.json b/schemas/testcaserunstarted.json index 854be0ea..91422b65 100644 --- a/schemas/testcaserunstarted.json +++ b/schemas/testcaserunstarted.json @@ -53,24 +53,6 @@ }, "content": { "properties": { - "type": { - "type": "string", - "enum": [ - "performance", - "functional", - "unit", - "security", - "compliance", - "integration", - "e2e", - "other" - ] - }, - "uri": { - "type": "string", - "format": "uri", - "description": "an optional URI to the actual testCase" - }, "trigger": { "type": "object", "description": "an optional definition of what triggered this testCase execution", @@ -132,9 +114,43 @@ "id" ] }, - "testCaseId": { - "type": "string", - "description": "an optional external identifier of the running testCase" + "testCase": { + "description": "the testCase which this testCaseRun is executing", + "properties": { + "id": { + "type": "string", + "minLength": 1 + }, + "source": { + "type": "string" + }, + "name": { + "type": "string" + }, + "type": { + "type": "string", + "enum": [ + "performance", + "functional", + "unit", + "security", + "compliance", + "integration", + "e2e", + "other" + ] + }, + "url": { + "type": "string", + "format": "uri" + } + }, + "additionalProperties": false, + "type": "object", + "required": [ + "id", + "source" + ] } }, "additionalProperties": false, diff --git a/schemas/testsuiterunfinished.json b/schemas/testsuiterunfinished.json index 04ae82a9..a6f35e49 100644 --- a/schemas/testsuiterunfinished.json +++ b/schemas/testsuiterunfinished.json @@ -78,9 +78,24 @@ "id" ] }, - "testSuiteId": { - "type": "string", - "description": "an optional external identifier of the testSuite execution" + "testSuite": { + "description": "the testSuite which this testSuiteRun is executing", + "properties": { + "id": { + "type": "string", + "minLength": 1 + }, + "source": { + "type": "string" + }, + "name": { + "type": "string" + }, + "url": { + "type": "string", + "format": "uri" + } + } }, "status": { "type": "string", diff --git a/schemas/testsuiterunqueued.json b/schemas/testsuiterunqueued.json index f8c7b684..ec635514 100644 --- a/schemas/testsuiterunqueued.json +++ b/schemas/testsuiterunqueued.json @@ -97,9 +97,24 @@ "id" ] }, - "testSuiteId": { - "type": "string", - "description": "an optional external identifier of the testSuite execution" + "testSuite": { + "description": "the testSuite which this testSuiteRun is executing", + "properties": { + "id": { + "type": "string", + "minLength": 1 + }, + "source": { + "type": "string" + }, + "name": { + "type": "string" + }, + "url": { + "type": "string", + "format": "uri" + } + } } }, "additionalProperties": false, diff --git a/schemas/testsuiterunstarted.json b/schemas/testsuiterunstarted.json index 9601fb02..c6a62347 100644 --- a/schemas/testsuiterunstarted.json +++ b/schemas/testsuiterunstarted.json @@ -97,9 +97,24 @@ "id" ] }, - "testSuiteId": { - "type": "string", - "description": "an optional external identifier of the testSuite execution" + "testSuite": { + "description": "the testSuite which this testSuiteRun is executing", + "properties": { + "id": { + "type": "string", + "minLength": 1 + }, + "source": { + "type": "string" + }, + "name": { + "type": "string" + }, + "url": { + "type": "string", + "format": "uri" + } + } } }, "additionalProperties": false, diff --git a/testing-events.md b/testing-events.md index 9bce912f..98b4de3c 100644 --- a/testing-events.md +++ b/testing-events.md @@ -26,41 +26,41 @@ This specification defines three subjects in this stage: `testCase`, `testSuite` ### `testCaseRun` -A `testCaseRun` is a process that performs a test against an input software artifact of some kind, for instance source code, a binary, a container image or else. +A `testCaseRun` is a process that executes a test against an input software artifact of some kind, for instance source code, a binary, a container image or else. A `testCaseRun` is the smallest unit of testing that the user wants to track, `testSuiteRuns` are for grouping purposes. -| Field | Type | Description | Examples | -|--------------|---------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------|---------------------------------------------------------------| -| id | `String` | Uniquely identifies the subject within the source. | `integration-test-abc`, `e2e-test1`, `scan-image1` | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | `staging/tekton`, `tekton-dev-123` | -| type | `String` | An optional type of test | `functional`, `unit`, `performance`, `security` | -| environment | `Object` [`environment`](continuous-deployment-pipeline-events.md/#environment) | The environment in which this testCaseRun is executing | `{"id": "1234"}`, `{"id": "dev", "source": "tekton-dev-123"}` | -| testCaseId | `String` | An optional testCase ID to enable collation of events related to a specific testCase definition | | | -| testSuiteRun | `Object` [`testSuiteRun`](#testsuiterun) | An optional testSuiteRun to associate this testCaseRun with a containing testSuiteRun | | | +| Field | Type | Description | Examples | +|--------------|---------------------------------------------------------------------------------|---------------------------------------------------------------------------------------|-------------------------------------------------------------------------| +| id | `String` | Uniquely identifies the subject within the source. | `Login-Test-execution-1213423`, `e2e-test1`, `2334234324` | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | `staging/testkube`, `testkube-dev-123` | +| type | `String` | An optional type of test | `functional`, `unit`, `performance`, `security` | +| environment | `Object` [`environment`](continuous-deployment-pipeline-events.md/#environment) | The environment in which this testCaseRun is executing | `{"id": "1234"}`, `{"id": "dev", "source": "testkube-dev-123"}` | +| testCase | `Object` [`testCase`](#testcase) | An optional definition of the testCase being executed | `{"id": "92834723894", "name": "Login Test", "type":"integration"}` | | +| testSuiteRun | `Object` [`testSuiteRun`](#testsuiterun) | An optional testSuiteRun to associate this testCaseRun with a containing testSuiteRun | `{"id":"Auth-TestSuite-execution-12334", "source": "staging/testkube"}` | | ### `testSuiteRun` -A `testSuiteRun` represents the execution of a grouped set of one or more `testCaseRuns`. +A `testSuiteRun` represents the execution of a set of one or more `testCaseRuns`. -| Field | Type | Description | Examples | -|-------------|---------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------|---------------------------------------------------------------| -| id | `String` | Uniquely identifies the subject within the source. | `my-testsuite`, `regression-123` | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | `staging/tekton`, `tekton-dev-123` | -| environment | `Object` [`environment`](continuous-deployment-pipeline-events.md/#environment) | The environment in which this testSuiteRun is executing | `{"id": "1234"}`, `{"id": "dev", "source": "tekton-dev-123"}` | -| testSuiteId | `String` | An optional testSuite ID to enable collation of events related to a specific testSuite definition | | | +| Field | Type | Description | Examples | +|-------------|---------------------------------------------------------------------------------|---------------------------------------------------------|-----------------------------------------------------------------| +| id | `String` | Uniquely identifies the subject within the source. | `Auth-TestSuite-execution-12334`, `regression-123` | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | `staging/testkube`, `testkube-dev-123` | +| environment | `Object` [`environment`](continuous-deployment-pipeline-events.md/#environment) | The environment in which this testSuiteRun is executing | `{"id": "1234"}`, `{"id": "dev", "source": "testkube-dev-123"}` | +| testSuite | `Object` [`testSuite`](#testsuite) | An optional definition of the testSuite being executed | `{"id": "92834723894", "name": "Auth TestSuite"}` | | ### `testOutput` One or more `testOutput` artifacts are usually produced as the result of a test execution. -| Field | Type | Description | Examples | -|-------------|----------------------------------------|---------------------------------------------------------------------------|----------------------------------------------------------| -| id | `String` | Uniquely identifies the subject within the source. | `23123123` | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | `staging/tekton`, `tekton-dev-123` | -| type | `String` | The type of output, one of `report`, `video`, `image`, `log`, `other` | `video` | -| format | `String` | The Content-Type of the output artifact | `application/pdf`, `image/png`, `application/json` | -| uri | `URI-Reference` | A reference to retrieve the specified output artifact | `https://testkube.mycluster.internal/artifacts/23123123` | -| testCaseRun | `Object` [`testCaseRun`](#testcaserun) | An optional testCaseRun to link this artifact to a specific `testCaseRun` | | | +| Field | Type | Description | Examples | +|-------------|----------------------------------------|---------------------------------------------------------------------------|-----------------------------------------------------------------------| +| id | `String` | Uniquely identifies the subject within the source. | `report-23123123` | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | `staging/testkube`, `testkube-dev-123` | +| type | `String` | The type of output, one of `report`, `video`, `image`, `log`, `other` | `video` | +| format | `String` | The Content-Type of the output artifact | `application/pdf`, `image/png`, `application/json` | +| uri | `URI-Reference` | A reference to retrieve the specified output artifact | `https://testkube.mycluster.internal/artifacts/23123123` | +| testCaseRun | `Object` [`testCaseRun`](#testcaserun) | An optional testCaseRun to link this artifact to a specific `testCaseRun` | `{"id":"Login-Test-execution-1213423", "source": "staging/testkube"}` | | ## Events @@ -73,15 +73,14 @@ This event represents when a testCaseRun has been queued for execution - and is - Predicate: queued - Subject: [`testCaseRun`](#testcaserun) -| Field | Type | Description | Examples | Required | -|--------------|---------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------|---------------------------------------------------------------|----------| -| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | -| type | `String` | An optional type of test | `functional`, `unit`, `performance`, `security` | -| environment | `Object` [`environment`](continuous-deployment-pipeline-events.md/#environment) | The environment in which this testCaseRun is queued | `{"id": "1234"}`, `{"id": "dev", "source": "tekton-dev-123"}` | ✅ | -| testCaseId | `String` | An optional external identifier to enable collation of events related to a specific testCase definition | | | -| testSuiteRun | `Object` [`testSuiteRun`](#testsuiterun) | An optional testSuiteRun to associate this testCaseRun with a containing testSuiteRun | | | -| trigger | `Object` [trigger](#trigger) | What triggered the queuing of this testCaseRun || +| Field | Type | Description | Examples | Required | +|--------------|---------------------------------------------------------------------------------|---------------------------------------------------------------------------------------|-------------------------------------------------------------------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| environment | `Object` [`environment`](continuous-deployment-pipeline-events.md/#environment) | The environment in which this testCaseRun is queued | `{"id": "1234"}`, `{"id": "dev", "source": "testkube-dev-123"}` | ✅ | +| testCase | `Object` [`testCase`](#testcase) | An optional definition of the testCase being executed | `{"id": "92834723894", "name": "Login Test", "type":"integration"}` | | +| testSuiteRun | `Object` [`testSuiteRun`](#testsuiterun) | An optional testSuiteRun to associate this testCaseRun with a containing testSuiteRun | `{"id":"Auth-TestSuite-execution-12334", "source": "staging/testkube"}` | | +| trigger | `Object` [trigger](#trigger) | What triggered this testSuiteRun | `{"type": "schedule}` | ### `testCaseRun started` @@ -91,15 +90,14 @@ This event represents a started testCase execution. - Predicate: started - Subject: [`testCaseRun`](#testcaserun) -| Field | Type | Description | Examples | Required | -|--------------|---------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------|---------------------------------------------------------------|----------| -| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | -| type | `String` | An optional type of test | `functional`, `unit`, `performance`, `security` | -| environment | `Object` [`environment`](continuous-deployment-pipeline-events.md/#environment) | The environment in which this testCaseRun is running | `{"id": "1234"}`, `{"id": "dev", "source": "tekton-dev-123"}` | ✅ | -| testCaseId | `String` | An optional external identifier to enable collation of events related to a specific testCase definition | | | -| testSuiteRun | `Object` [`testSuiteRun`](#testsuiterun) | An optional testSuiteRun to associate this testCaseRun with a containing testSuiteRun | | | -| trigger | `Object` [trigger](#trigger) | What triggered this testCaseRun || +| Field | Type | Description | Examples | Required | +|--------------|---------------------------------------------------------------------------------|---------------------------------------------------------------------------------------|-------------------------------------------------------------------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| environment | `Object` [`environment`](continuous-deployment-pipeline-events.md/#environment) | The environment in which this testCaseRun is running | `{"id": "1234"}`, `{"id": "dev", "source": "testkube-dev-123"}` | ✅ | +| testCase | `Object` [`testCase`](#testcase) | An optional definition of the testCase being executed | `{"id": "92834723894", "name": "Login Test", "type":"integration"}` | | +| testSuiteRun | `Object` [`testSuiteRun`](#testsuiterun) | An optional testSuiteRun to associate this testCaseRun with a containing testSuiteRun | `{"id":"Auth-TestSuite-execution-12334", "source": "staging/testkube"}` | | +| trigger | `Object` [trigger](#trigger) | What triggered this testSuiteRun | `{"type": "event}` | ### `testCaseRun finished` @@ -109,17 +107,17 @@ This event represents a finished testCase execution. The event will contain the - Predicate: finished - Subject: [`testCaseRun`](#testcaserun) -| Field | Type | Description | Examples | Required | -|--------------|---------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------|---------------------------------------------------------------|----------| -| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | -| type | `String` | An optional type of test | `functional`, `unit`, `performance`, `security` | -| environment | `Object` [`environment`](continuous-deployment-pipeline-events.md/#environment) | The environment in which this testCaseRun was running | `{"id": "1234"}`, `{"id": "dev", "source": "tekton-dev-123"}` | ✅ | -| testCaseId | `String` | An optional external identifier to enable collation of events related to a specific testCase definition | | | -| testSuiteRun | `Object` [`testSuiteRun`](#testsuiterun) | An optional testSuiteRun to associate this testCaseRun with a containing testSuiteRun | | | -| status | `String` | The status of the testSuite execution, one of `pass`, `fail`, `cancel`, `error` | | ✅ | -| severity | `String` | An optional severity if the test failed, one of `low`, `medium`, `high`, `critical` | `critical`, `low`, `medium`, `high` | -| reason | `String` | An optional reason related to the `status` of the execution | `Cancelled by user`, `Failed assertion`, `Timed out` | | +| Field | Type | Description | Examples | Required | +|--------------|---------------------------------------------------------------------------------|---------------------------------------------------------------------------------------|-------------------------------------------------------------------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| type | `String` | An optional type of test | `functional`, `unit`, `performance`, `security` | +| environment | `Object` [`environment`](continuous-deployment-pipeline-events.md/#environment) | The environment in which this testCaseRun was running | `{"id": "1234"}`, `{"id": "dev", "source": "testkube-dev-123"}` | ✅ | +| testCase | `Object` [`testCase`](#testcase) | An optional definition of the testCase being executed | `{"id": "92834723894", "name": "Login Test", "type":"integration"}` | | +| testSuiteRun | `Object` [`testSuiteRun`](#testsuiterun) | An optional testSuiteRun to associate this testCaseRun with a containing testSuiteRun | `{"id":"Auth-TestSuite-execution-12334", "source": "staging/testkube"}` | | +| status | `String` | The status of the testSuite execution, one of `pass`, `fail`, `cancel`, `error` | `pass` | ✅ | +| severity | `String` | An optional severity if the test failed, one of `low`, `medium`, `high`, `critical` | `critical` | +| reason | `String` | An optional reason related to the `status` of the execution | `Cancelled by user`, `Failed assertion`, `Timed out` | | ### `testSuiteRun queued` @@ -130,13 +128,13 @@ This event represents when a testSuiteRun has been queued for execution - and is - Predicate: queued - Subject: [`testSuiteRun`](#testsuiterun) -| Field | Type | Description | Examples | Required | -|-------------|---------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------|---------------------------------------------------------------|----------| -| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | -| environment | `Object` [`environment`](continuous-deployment-pipeline-events.md/#environment) | The environment in which this testSuiteRun is queued | `{"id": "1234"}`, `{"id": "dev", "source": "tekton-dev-123"}` | ✅ | -| testSuiteId | `String` | An optional external identifier to enable collation of events related to a specific testSuite definition | | | -| trigger | `Object` [trigger](#trigger) | What triggered this testSuiteRun || +| Field | Type | Description | Examples | Required | +|-------------|---------------------------------------------------------------------------------|--------------------------------------------------------|-----------------------------------------------------------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| environment | `Object` [`environment`](continuous-deployment-pipeline-events.md/#environment) | The environment in which this testSuiteRun is queued | `{"id": "1234"}`, `{"id": "dev", "source": "testkube-dev-123"}` | ✅ | +| testSuite | `Object` [`testSuite`](#testsuite) | An optional definition of the testSuite being executed | `{"id": "92834723894", "name": "Auth TestSuite"}` | | +| trigger | `Object` [trigger](#trigger) | What triggered this testSuiteRun | `{"type": "manual}` | ### `testSuiteRun started` @@ -146,13 +144,13 @@ This event represents a started testSuite execution. - Predicate: started - Subject: [`testSuiteRun`](#testsuiterun) -| Field | Type | Description | Examples | Required | -|-------------|---------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------|---------------------------------------------------------------|----------| -| id | `String` | Uniquely identifies the subject within the source. | `unit`, `e2e`, `security` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | -| environment | `Object` [`environment`](continuous-deployment-pipeline-events.md/#environment) | The environment in which this testSuiteRun is running | `{"id": "1234"}`, `{"id": "dev", "source": "tekton-dev-123"}` | ✅ | -| testSuiteId | `String` | An optional external identifier to enable collation of events related to a specific testSuite definition | | | -| trigger | `Object` [trigger](#trigger) | What triggered this testSuiteRun || +| Field | Type | Description | Examples | Required | +|-------------|---------------------------------------------------------------------------------|--------------------------------------------------------|-----------------------------------------------------------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `unit`, `e2e`, `security` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| environment | `Object` [`environment`](continuous-deployment-pipeline-events.md/#environment) | The environment in which this testSuiteRun is running | `{"id": "1234"}`, `{"id": "dev", "source": "testkube-dev-123"}` | ✅ | +| testSuite | `Object` [`testSuite`](#testsuite) | An optional definition of the testSuite being executed | `{"id": "92834723894", "name": "Auth TestSuite"}` | | +| trigger | `Object` [trigger](#trigger) | What triggered this testSuiteRun | `{"type": "pipeline}` | ### `testSuiteRun finished` @@ -162,15 +160,15 @@ This event represents a finished testSuite execution. The event will contain the - Predicate: finished - Subject: [`testSuiteRun`](#testsuiterun) -| Field | Type | Description | Examples | Required | -|-------------|---------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------|---------------------------------------------------------------|----------| -| id | `String` | Uniquely identifies the subject within the source. | `unit`, `e2e`, `security` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | -| environment | `Object` [`environment`](continuous-deployment-pipeline-events.md/#environment) | The environment in which this testSuiteRun was running | `{"id": "1234"}`, `{"id": "dev", "source": "tekton-dev-123"}` | ✅ | -| testSuiteId | `String` | An optional testSuite ID to enable collation of events related to a specific testSuite definition | | | -| status | `String` | The status of the testSuite execution, one of `pass`, `fail`, `cancel`, `error` | | ✅ | -| severity | `String` | An optional severity if the test failed, one of `low`, `medium`, `high`, `critical` | `critical`, `low`, `medium`, `high` | -| reason | `String` | An optional reason related to the status of the execution | `Cancelled by user`, `Failed testCase` | | +| Field | Type | Description | Examples | Required | +|-------------|---------------------------------------------------------------------------------|-------------------------------------------------------------------------------------|-----------------------------------------------------------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `unit`, `e2e`, `security` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| environment | `Object` [`environment`](continuous-deployment-pipeline-events.md/#environment) | The environment in which this testSuiteRun was running | `{"id": "1234"}`, `{"id": "dev", "source": "testkube-dev-123"}` | ✅ | +| status | `String` | The status of the testSuite execution, one of `pass`, `fail`, `cancel`, `error` | `fail` | ✅ | +| severity | `String` | An optional severity if the test failed, one of `low`, `medium`, `high`, `critical` | `critical`, `low`, `medium`, `high` | +| reason | `String` | An optional reason related to the status of the execution | `Cancelled by user`, `Failed testCase` | | +| testSuite | `Object` [`testSuite`](#testsuite) | An optional definition of the testSuite being executed | `{"id": "92834723894", "name": "Auth TestSuite"}` | | ### `testOutput published` @@ -180,14 +178,14 @@ The event represents a test execution output artifact that has been published. - Predicate: published - Subject: [`testOutput`](#testoutput) -| Field | Type | Description | Examples | Required | -|-------------|----------------------------------------|---------------------------------------------------------------------------|----------------------------------------------------------|----------| -| id | `String` | Uniquely identifies the subject within the source. | `12312334` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | ✅ | -| type | `String` | The type of output, one of `report`, `video`, `image`, `log`, `other` | `video` | ✅ | -| format | `String` | The Content-Type of the output artifact | `application/pdf`, `image/png`, `application/json` | ✅ | -| uri | `URI-Reference` | A reference to retrieve the specified output artifact | `https://testkube.mycluster.internal/artifacts/23123123` | ✅ | -| testCaseRun | `Object` [`testCaseRun`](#testcaserun) | An optional testCaseRun to link this artifact to a specific `testCaseRun` | | | +| Field | Type | Description | Examples | Required | +|-------------|----------------------------------------|---------------------------------------------------------------------------|-----------------------------------------------------------------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `12312334` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | ✅ | +| type | `String` | The type of output, one of `report`, `video`, `image`, `log`, `other` | `video` | ✅ | +| format | `String` | The Content-Type of the output artifact | `application/pdf`, `image/png`, `application/json` | ✅ | +| uri | `URI-Reference` | A reference to retrieve the specified output artifact | `https://testkube.mycluster.internal/artifacts/23123123` | ✅ | +| testCaseRun | `Object` [`testCaseRun`](#testcaserun) | An optional testCaseRun to link this artifact to a specific `testCaseRun` | `{"id":"Login-Test-execution-1213423", "source": "staging/testkube"}` | | ## Common Objects @@ -200,3 +198,28 @@ A `trigger` in this context is what started a corresponding testCaseRun/testSuit | type | `String` | The type of trigger, one of `manual`, `pipeline`, `event`, `schedule`, `other` | | ✅ | | uri | `URI-Reference` | An optional uri reference to this trigger | | | +### `testCase` + +A `testCase` is the actual test that is being run by a `testCaseRun`. + +| Field | Type | Description | Examples | Required | +|--------|-----------------|-----------------------------------------------------|----------------------------|----------| +| id | `String` | Uniquely identifies the testCase within the source. | `12312334` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | ✅ | +| type | `String` | An optional type of test | `functional`,`performance` | | +| name | `String` | An user-friendly name for this testCase | `Login Test` | | +| uri | `URI-Reference` | An optional uri reference to this testCase | | | + + +### `testSuite` + +A `testSuite` is a collection of `testCase` objects managed in an external system. Each time a `testSuite` is executed the +corresponding `testSuiteXXX` events should be emmitted. + +| Field | Type | Description | Examples | Required | +|--------|-----------------|------------------------------------------------------|------------------|----------| +| id | `String` | Uniquely identifies the testSuite within the source. | `12312334` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | ✅ | +| name | `String` | An user-friendly name for this testSuite | `Auth TestSuite` | | +| uri | `URI-Reference` | An optional uri reference to this testSuite | | | + From 6a8bd2ab63783071e5f768c2ea7bc70cc251a4cf Mon Sep 17 00:00:00 2001 From: olensmar Date: Thu, 30 Mar 2023 14:39:17 +0200 Subject: [PATCH 19/37] added examples and fixed errors --- examples/testcaserun_finished.json | 26 +++++ examples/testcaserun_queued.json | 28 ++++++ examples/testcaserun_started.json | 28 ++++++ examples/testoutput_published.json | 22 +++++ examples/testsuiterun_finished.json | 27 ++++++ examples/testsuiterun_queued.json | 27 ++++++ examples/testsuiterun_started.json | 27 ++++++ schemas/testoutputpublished.json | 2 +- testing-events.md | 145 ++++++++++++++-------------- 9 files changed, 258 insertions(+), 74 deletions(-) create mode 100644 examples/testcaserun_finished.json create mode 100644 examples/testcaserun_queued.json create mode 100644 examples/testcaserun_started.json create mode 100644 examples/testoutput_published.json create mode 100644 examples/testsuiterun_finished.json create mode 100644 examples/testsuiterun_queued.json create mode 100644 examples/testsuiterun_started.json diff --git a/examples/testcaserun_finished.json b/examples/testcaserun_finished.json new file mode 100644 index 00000000..29c0e3ba --- /dev/null +++ b/examples/testcaserun_finished.json @@ -0,0 +1,26 @@ +{ + "context": { + "version": "0.2.0", + "id": "271069a8-fc18-44f1-b38f-9d70a1695819", + "source": "/event/source/123", + "type": "dev.cdevents.testcaserun.started.0.1.0-draft", + "timestamp": "2023-03-20T14:27:05.315384Z" + }, + "subject": { + "id": "myTestCaseRun123", + "source": "/event/source/123", + "type": "testCaseRun", + "content": { + "status" : "pass", + "environment": { + "id": "dev", + "source": "testkube-dev-123" + }, + "testCase": { + "id": "92834723894", + "name": "Login Test", + "type": "integration" + } + } + } +} diff --git a/examples/testcaserun_queued.json b/examples/testcaserun_queued.json new file mode 100644 index 00000000..e6e311ee --- /dev/null +++ b/examples/testcaserun_queued.json @@ -0,0 +1,28 @@ +{ + "context": { + "version": "0.2.0", + "id": "271069a8-fc18-44f1-b38f-9d70a1695819", + "source": "/event/source/123", + "type": "dev.cdevents.testcaserun.queued.0.1.0-draft", + "timestamp": "2023-03-20T14:27:05.315384Z" + }, + "subject": { + "id": "myTestCaseRun123", + "source": "/event/source/123", + "type": "testCaseRun", + "content": { + "environment": { + "id": "dev", + "source": "testkube-dev-123" + }, + "testCase": { + "id": "92834723894", + "name": "Login Test", + "type": "integration" + }, + "trigger": { + "type": "schedule" + } + } + } +} diff --git a/examples/testcaserun_started.json b/examples/testcaserun_started.json new file mode 100644 index 00000000..dc959827 --- /dev/null +++ b/examples/testcaserun_started.json @@ -0,0 +1,28 @@ +{ + "context": { + "version": "0.2.0", + "id": "271069a8-fc18-44f1-b38f-9d70a1695819", + "source": "/event/source/123", + "type": "dev.cdevents.testcaserun.started.0.1.0-draft", + "timestamp": "2023-03-20T14:27:05.315384Z" + }, + "subject": { + "id": "myTestCaseRun123", + "source": "/event/source/123", + "type": "testCaseRun", + "content": { + "environment": { + "id": "dev", + "source": "testkube-dev-123" + }, + "testCase": { + "id": "92834723894", + "name": "Login Test", + "type": "integration" + }, + "trigger": { + "type": "schedule" + } + } + } +} diff --git a/examples/testoutput_published.json b/examples/testoutput_published.json new file mode 100644 index 00000000..23137740 --- /dev/null +++ b/examples/testoutput_published.json @@ -0,0 +1,22 @@ +{ + "context": { + "version": "0.2.0", + "id": "271069a8-fc18-44f1-b38f-9d70a1695819", + "source": "/event/source/123", + "type": "dev.cdevents.testoutput.published.0.1.0-draft", + "timestamp": "2023-03-20T14:27:05.315384Z" + }, + "subject": { + "id": "testrunreport-12123", + "source": "/event/source/testrunreport-12123", + "type": "testOutput", + "content": { + "outputType" : "video", + "format" : "video/quicktime", + "testCaseRun": { + "id": "myTestCaseRun123", + "source": "/event/source/123" + } + } + } +} diff --git a/examples/testsuiterun_finished.json b/examples/testsuiterun_finished.json new file mode 100644 index 00000000..7ba06aa6 --- /dev/null +++ b/examples/testsuiterun_finished.json @@ -0,0 +1,27 @@ +{ + "context": { + "version": "0.2.0", + "id": "271069a8-fc18-44f1-b38f-9d70a1695819", + "source": "/event/source/123", + "type": "dev.cdevents.testsuiterun.finished.0.1.0-draft", + "timestamp": "2023-03-20T14:27:05.315384Z" + }, + "subject": { + "id": "myTestSuiteRun123", + "source": "/event/source/123", + "type": "testSuiteRun", + "content": { + "status" : "failed", + "severity" : "critical", + "reason" : "Host 123.34.23.32 not found", + "environment": { + "id": "dev", + "source": "testkube-dev-123" + }, + "testSuite": { + "id": "92834723894", + "name": "Auth TestSuite" + } + } + } +} diff --git a/examples/testsuiterun_queued.json b/examples/testsuiterun_queued.json new file mode 100644 index 00000000..bd377804 --- /dev/null +++ b/examples/testsuiterun_queued.json @@ -0,0 +1,27 @@ +{ + "context": { + "version": "0.2.0", + "id": "271069a8-fc18-44f1-b38f-9d70a1695819", + "source": "/event/source/123", + "type": "dev.cdevents.testsuiterun.queued.0.1.0-draft", + "timestamp": "2023-03-20T14:27:05.315384Z" + }, + "subject": { + "id": "myTestSuiteRun123", + "source": "/event/source/123", + "type": "testSuiteRun", + "content": { + "environment": { + "id": "dev", + "source": "testkube-dev-123" + }, + "testSuite": { + "id": "92834723894", + "name": "Auth TestSuite" + }, + "trigger": { + "type": "pipeline" + } + } + } +} diff --git a/examples/testsuiterun_started.json b/examples/testsuiterun_started.json new file mode 100644 index 00000000..8b96855e --- /dev/null +++ b/examples/testsuiterun_started.json @@ -0,0 +1,27 @@ +{ + "context": { + "version": "0.2.0", + "id": "271069a8-fc18-44f1-b38f-9d70a1695819", + "source": "/event/source/123", + "type": "dev.cdevents.testsuiterun.started.0.1.0-draft", + "timestamp": "2023-03-20T14:27:05.315384Z" + }, + "subject": { + "id": "myTestSuiteRun123", + "source": "/event/source/123", + "type": "testSuiteRun", + "content": { + "environment": { + "id": "dev", + "source": "testkube-dev-123" + }, + "testSuite": { + "id": "92834723894", + "name": "Auth TestSuite" + }, + "trigger": { + "type": "pipeline" + } + } + } +} diff --git a/schemas/testoutputpublished.json b/schemas/testoutputpublished.json index deb1b1d4..63e5cf4a 100644 --- a/schemas/testoutputpublished.json +++ b/schemas/testoutputpublished.json @@ -54,7 +54,7 @@ "content": { "properties": { "type": { - "type": "string", + "outputType": "string", "enum": [ "report", "video", diff --git a/testing-events.md b/testing-events.md index 98b4de3c..9df4344e 100644 --- a/testing-events.md +++ b/testing-events.md @@ -10,13 +10,13 @@ description: > --> # Testing Events -Testing events covers the subjects and predicates related to test-execution performed either independently or as part of CI/CD pipelines. +Testing events covers the subjects and predicates related to test-execution performed either independently or as part of CI/CD pipelines. > These subjects replace the previously defined testCase/testSuite subjects in the Continuous Integration category ## Subjects -This specification defines three subjects in this stage: `testCase`, `testSuite` and `testOutput`. +This specification defines three subjects in this stage: `testCase`, `testSuite` and `testOutput`. | Subject | Description | Predicates | |---------------------------------|----------------------------------------------|------------------------------------------------------------------------------------------------------------| @@ -29,25 +29,25 @@ This specification defines three subjects in this stage: `testCase`, `testSuite` A `testCaseRun` is a process that executes a test against an input software artifact of some kind, for instance source code, a binary, a container image or else. A `testCaseRun` is the smallest unit of testing that the user wants to track, `testSuiteRuns` are for grouping purposes. -| Field | Type | Description | Examples | -|--------------|---------------------------------------------------------------------------------|---------------------------------------------------------------------------------------|-------------------------------------------------------------------------| -| id | `String` | Uniquely identifies the subject within the source. | `Login-Test-execution-1213423`, `e2e-test1`, `2334234324` | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | `staging/testkube`, `testkube-dev-123` | -| type | `String` | An optional type of test | `functional`, `unit`, `performance`, `security` | -| environment | `Object` [`environment`](continuous-deployment-pipeline-events.md/#environment) | The environment in which this testCaseRun is executing | `{"id": "1234"}`, `{"id": "dev", "source": "testkube-dev-123"}` | -| testCase | `Object` [`testCase`](#testcase) | An optional definition of the testCase being executed | `{"id": "92834723894", "name": "Login Test", "type":"integration"}` | | -| testSuiteRun | `Object` [`testSuiteRun`](#testsuiterun) | An optional testSuiteRun to associate this testCaseRun with a containing testSuiteRun | `{"id":"Auth-TestSuite-execution-12334", "source": "staging/testkube"}` | | +| Field | Type | Description | Examples | +|--------------|-----------------------------------------------------------------|---------------------------------------------------------------------------------------|-------------------------------------------------------------------------| +| id | `String` | Uniquely identifies the subject within the source. | `Login-Test-execution-1213423`, `e2e-test1`, `2334234324` | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | `staging/testkube`, `testkube-dev-123` | +| type | `String` | An optional type of test | `functional`, `unit`, `performance`, `security` | +| environment | `Object` [`environment`](continuous-deployment.md/#environment) | The environment in which this testCaseRun is executing | `{"id": "1234"}`, `{"id": "dev", "source": "testkube-dev-123"}` | +| testCase | `Object` [`testCase`](#testcase) | An optional definition of the testCase being executed | `{"id": "92834723894", "name": "Login Test", "type":"integration"}` | | +| testSuiteRun | `Object` [`testSuiteRun`](#testsuiterun) | An optional testSuiteRun to associate this testCaseRun with a containing testSuiteRun | `{"id":"Auth-TestSuite-execution-12334", "source": "staging/testkube"}` | | ### `testSuiteRun` A `testSuiteRun` represents the execution of a set of one or more `testCaseRuns`. -| Field | Type | Description | Examples | -|-------------|---------------------------------------------------------------------------------|---------------------------------------------------------|-----------------------------------------------------------------| -| id | `String` | Uniquely identifies the subject within the source. | `Auth-TestSuite-execution-12334`, `regression-123` | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | `staging/testkube`, `testkube-dev-123` | -| environment | `Object` [`environment`](continuous-deployment-pipeline-events.md/#environment) | The environment in which this testSuiteRun is executing | `{"id": "1234"}`, `{"id": "dev", "source": "testkube-dev-123"}` | -| testSuite | `Object` [`testSuite`](#testsuite) | An optional definition of the testSuite being executed | `{"id": "92834723894", "name": "Auth TestSuite"}` | | +| Field | Type | Description | Examples | +|-------------|-----------------------------------------------------------------|---------------------------------------------------------|-----------------------------------------------------------------| +| id | `String` | Uniquely identifies the subject within the source. | `Auth-TestSuite-execution-12334`, `regression-123` | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | `staging/testkube`, `testkube-dev-123` | +| environment | `Object` [`environment`](continuous-deployment.md/#environment) | The environment in which this testSuiteRun is executing | `{"id": "1234"}`, `{"id": "dev", "source": "testkube-dev-123"}` | +| testSuite | `Object` [`testSuite`](#testsuite) | An optional definition of the testSuite being executed | `{"id": "92834723894", "name": "Auth TestSuite"}` | | ### `testOutput` @@ -58,9 +58,9 @@ One or more `testOutput` artifacts are usually produced as the result of a test | id | `String` | Uniquely identifies the subject within the source. | `report-23123123` | | source | `URI-Reference` | [source](spec.md#source--context-) from the context | `staging/testkube`, `testkube-dev-123` | | type | `String` | The type of output, one of `report`, `video`, `image`, `log`, `other` | `video` | -| format | `String` | The Content-Type of the output artifact | `application/pdf`, `image/png`, `application/json` | -| uri | `URI-Reference` | A reference to retrieve the specified output artifact | `https://testkube.mycluster.internal/artifacts/23123123` | -| testCaseRun | `Object` [`testCaseRun`](#testcaserun) | An optional testCaseRun to link this artifact to a specific `testCaseRun` | `{"id":"Login-Test-execution-1213423", "source": "staging/testkube"}` | | +| format | `String` | The Content-Type of the output artifact | `application/pdf`, `image/png`, `application/json` | +| uri | `URI-Reference` | A reference to retrieve the specified output artifact | `https://testkube.mycluster.internal/artifacts/23123123` | +| testCaseRun | `Object` [`testCaseRun`](#testcaserun) | An optional testCaseRun to link this artifact to a specific `testCaseRun` | `{"id":"Login-Test-execution-1213423", "source": "staging/testkube"}` | ## Events @@ -73,31 +73,31 @@ This event represents when a testCaseRun has been queued for execution - and is - Predicate: queued - Subject: [`testCaseRun`](#testcaserun) -| Field | Type | Description | Examples | Required | -|--------------|---------------------------------------------------------------------------------|---------------------------------------------------------------------------------------|-------------------------------------------------------------------------|----------| -| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | -| environment | `Object` [`environment`](continuous-deployment-pipeline-events.md/#environment) | The environment in which this testCaseRun is queued | `{"id": "1234"}`, `{"id": "dev", "source": "testkube-dev-123"}` | ✅ | -| testCase | `Object` [`testCase`](#testcase) | An optional definition of the testCase being executed | `{"id": "92834723894", "name": "Login Test", "type":"integration"}` | | -| testSuiteRun | `Object` [`testSuiteRun`](#testsuiterun) | An optional testSuiteRun to associate this testCaseRun with a containing testSuiteRun | `{"id":"Auth-TestSuite-execution-12334", "source": "staging/testkube"}` | | -| trigger | `Object` [trigger](#trigger) | What triggered this testSuiteRun | `{"type": "schedule}` | +| Field | Type | Description | Examples | Required | +|--------------|-----------------------------------------------------------------|---------------------------------------------------------------------------------------|-------------------------------------------------------------------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| environment | `Object` [`environment`](continuous-deployment.md/#environment) | The environment in which this testCaseRun is queued | `{"id": "1234"}`, `{"id": "dev", "source": "testkube-dev-123"}` | ✅ | +| testCase | `Object` [`testCase`](#testcase) | An optional definition of the testCase being executed | `{"id": "92834723894", "name": "Login Test", "type":"integration"}` | | +| testSuiteRun | `Object` [`testSuiteRun`](#testsuiterun) | An optional testSuiteRun to associate this testCaseRun with a containing testSuiteRun | `{"id":"Auth-TestSuite-execution-12334", "source": "staging/testkube"}` | | +| trigger | `Object` [trigger](#trigger) | What triggered this testSuiteRun | `{"type": "schedule"}` | ### `testCaseRun started` -This event represents a started testCase execution. +This event represents a started testCase execution. - Event Type: __`dev.cdevents.testcaserun.started.0.1.0-draft`__ - Predicate: started - Subject: [`testCaseRun`](#testcaserun) -| Field | Type | Description | Examples | Required | -|--------------|---------------------------------------------------------------------------------|---------------------------------------------------------------------------------------|-------------------------------------------------------------------------|----------| -| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | -| environment | `Object` [`environment`](continuous-deployment-pipeline-events.md/#environment) | The environment in which this testCaseRun is running | `{"id": "1234"}`, `{"id": "dev", "source": "testkube-dev-123"}` | ✅ | -| testCase | `Object` [`testCase`](#testcase) | An optional definition of the testCase being executed | `{"id": "92834723894", "name": "Login Test", "type":"integration"}` | | -| testSuiteRun | `Object` [`testSuiteRun`](#testsuiterun) | An optional testSuiteRun to associate this testCaseRun with a containing testSuiteRun | `{"id":"Auth-TestSuite-execution-12334", "source": "staging/testkube"}` | | -| trigger | `Object` [trigger](#trigger) | What triggered this testSuiteRun | `{"type": "event}` | +| Field | Type | Description | Examples | Required | +|--------------|-----------------------------------------------------------------|---------------------------------------------------------------------------------------|-------------------------------------------------------------------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| environment | `Object` [`environment`](continuous-deployment.md/#environment) | The environment in which this testCaseRun is running | `{"id": "1234"}`, `{"id": "dev", "source": "testkube-dev-123"}` | ✅ | +| testCase | `Object` [`testCase`](#testcase) | An optional definition of the testCase being executed | `{"id": "92834723894", "name": "Login Test", "type":"integration"}` | | +| testSuiteRun | `Object` [`testSuiteRun`](#testsuiterun) | An optional testSuiteRun to associate this testCaseRun with a containing testSuiteRun | `{"id":"Auth-TestSuite-execution-12334", "source": "staging/testkube"}` | | +| trigger | `Object` [trigger](#trigger) | What triggered this testSuiteRun | `{"type": "event"}` | ### `testCaseRun finished` @@ -107,17 +107,16 @@ This event represents a finished testCase execution. The event will contain the - Predicate: finished - Subject: [`testCaseRun`](#testcaserun) -| Field | Type | Description | Examples | Required | -|--------------|---------------------------------------------------------------------------------|---------------------------------------------------------------------------------------|-------------------------------------------------------------------------|----------| -| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | -| type | `String` | An optional type of test | `functional`, `unit`, `performance`, `security` | -| environment | `Object` [`environment`](continuous-deployment-pipeline-events.md/#environment) | The environment in which this testCaseRun was running | `{"id": "1234"}`, `{"id": "dev", "source": "testkube-dev-123"}` | ✅ | -| testCase | `Object` [`testCase`](#testcase) | An optional definition of the testCase being executed | `{"id": "92834723894", "name": "Login Test", "type":"integration"}` | | -| testSuiteRun | `Object` [`testSuiteRun`](#testsuiterun) | An optional testSuiteRun to associate this testCaseRun with a containing testSuiteRun | `{"id":"Auth-TestSuite-execution-12334", "source": "staging/testkube"}` | | -| status | `String` | The status of the testSuite execution, one of `pass`, `fail`, `cancel`, `error` | `pass` | ✅ | -| severity | `String` | An optional severity if the test failed, one of `low`, `medium`, `high`, `critical` | `critical` | -| reason | `String` | An optional reason related to the `status` of the execution | `Cancelled by user`, `Failed assertion`, `Timed out` | | +| Field | Type | Description | Examples | Required | +|--------------|-----------------------------------------------------------------|---------------------------------------------------------------------------------------|-------------------------------------------------------------------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| environment | `Object` [`environment`](continuous-deployment.md/#environment) | The environment in which this testCaseRun was running | `{"id": "1234"}`, `{"id": "dev", "source": "testkube-dev-123"}` | ✅ | +| testCase | `Object` [`testCase`](#testcase) | An optional definition of the testCase being executed | `{"id": "92834723894", "name": "Login Test", "type":"integration"}` | | +| testSuiteRun | `Object` [`testSuiteRun`](#testsuiterun) | An optional testSuiteRun to associate this testCaseRun with a containing testSuiteRun | `{"id":"Auth-TestSuite-execution-12334", "source": "staging/testkube"}` | | +| status | `String` | The status of the testSuite execution, one of `pass`, `fail`, `cancel`, `error` | `pass` | ✅ | +| severity | `String` | An optional severity if the test failed, one of `low`, `medium`, `high`, `critical` | `critical` | +| reason | `String` | An optional reason related to the `status` of the execution | `Cancelled by user`, `Failed assertion`, `Timed out` | | ### `testSuiteRun queued` @@ -128,13 +127,13 @@ This event represents when a testSuiteRun has been queued for execution - and is - Predicate: queued - Subject: [`testSuiteRun`](#testsuiterun) -| Field | Type | Description | Examples | Required | -|-------------|---------------------------------------------------------------------------------|--------------------------------------------------------|-----------------------------------------------------------------|----------| -| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | -| environment | `Object` [`environment`](continuous-deployment-pipeline-events.md/#environment) | The environment in which this testSuiteRun is queued | `{"id": "1234"}`, `{"id": "dev", "source": "testkube-dev-123"}` | ✅ | -| testSuite | `Object` [`testSuite`](#testsuite) | An optional definition of the testSuite being executed | `{"id": "92834723894", "name": "Auth TestSuite"}` | | -| trigger | `Object` [trigger](#trigger) | What triggered this testSuiteRun | `{"type": "manual}` | +| Field | Type | Description | Examples | Required | +|-------------|-----------------------------------------------------------------|--------------------------------------------------------|-----------------------------------------------------------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| environment | `Object` [`environment`](continuous-deployment.md/#environment) | The environment in which this testSuiteRun is queued | `{"id": "1234"}`, `{"id": "dev", "source": "testkube-dev-123"}` | ✅ | +| testSuite | `Object` [`testSuite`](#testsuite) | An optional definition of the testSuite being executed | `{"id": "92834723894", "name": "Auth TestSuite"}` | | +| trigger | `Object` [trigger](#trigger) | What triggered this testSuiteRun | `{"type": "manual"}` | ### `testSuiteRun started` @@ -144,13 +143,13 @@ This event represents a started testSuite execution. - Predicate: started - Subject: [`testSuiteRun`](#testsuiterun) -| Field | Type | Description | Examples | Required | -|-------------|---------------------------------------------------------------------------------|--------------------------------------------------------|-----------------------------------------------------------------|----------| -| id | `String` | Uniquely identifies the subject within the source. | `unit`, `e2e`, `security` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | -| environment | `Object` [`environment`](continuous-deployment-pipeline-events.md/#environment) | The environment in which this testSuiteRun is running | `{"id": "1234"}`, `{"id": "dev", "source": "testkube-dev-123"}` | ✅ | -| testSuite | `Object` [`testSuite`](#testsuite) | An optional definition of the testSuite being executed | `{"id": "92834723894", "name": "Auth TestSuite"}` | | -| trigger | `Object` [trigger](#trigger) | What triggered this testSuiteRun | `{"type": "pipeline}` | +| Field | Type | Description | Examples | Required | +|-------------|-----------------------------------------------------------------|--------------------------------------------------------|-----------------------------------------------------------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `unit`, `e2e`, `security` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| environment | `Object` [`environment`](continuous-deployment.md/#environment) | The environment in which this testSuiteRun is running | `{"id": "1234"}`, `{"id": "dev", "source": "testkube-dev-123"}` | ✅ | +| testSuite | `Object` [`testSuite`](#testsuite) | An optional definition of the testSuite being executed | `{"id": "92834723894", "name": "Auth TestSuite"}` | | +| trigger | `Object` [trigger](#trigger) | What triggered this testSuiteRun | `{"type": "pipeline"}` | ### `testSuiteRun finished` @@ -160,15 +159,15 @@ This event represents a finished testSuite execution. The event will contain the - Predicate: finished - Subject: [`testSuiteRun`](#testsuiterun) -| Field | Type | Description | Examples | Required | -|-------------|---------------------------------------------------------------------------------|-------------------------------------------------------------------------------------|-----------------------------------------------------------------|----------| -| id | `String` | Uniquely identifies the subject within the source. | `unit`, `e2e`, `security` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | -| environment | `Object` [`environment`](continuous-deployment-pipeline-events.md/#environment) | The environment in which this testSuiteRun was running | `{"id": "1234"}`, `{"id": "dev", "source": "testkube-dev-123"}` | ✅ | -| status | `String` | The status of the testSuite execution, one of `pass`, `fail`, `cancel`, `error` | `fail` | ✅ | -| severity | `String` | An optional severity if the test failed, one of `low`, `medium`, `high`, `critical` | `critical`, `low`, `medium`, `high` | -| reason | `String` | An optional reason related to the status of the execution | `Cancelled by user`, `Failed testCase` | | -| testSuite | `Object` [`testSuite`](#testsuite) | An optional definition of the testSuite being executed | `{"id": "92834723894", "name": "Auth TestSuite"}` | | +| Field | Type | Description | Examples | Required | +|-------------|-----------------------------------------------------------------|-------------------------------------------------------------------------------------|-----------------------------------------------------------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `unit`, `e2e`, `security` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| environment | `Object` [`environment`](continuous-deployment.md/#environment) | The environment in which this testSuiteRun was running | `{"id": "1234"}`, `{"id": "dev", "source": "testkube-dev-123"}` | ✅ | +| status | `String` | The status of the testSuite execution, one of `pass`, `fail`, `cancel`, `error` | `fail` | ✅ | +| severity | `String` | An optional severity if the test failed, one of `low`, `medium`, `high`, `critical` | `critical`, `low`, `medium`, `high` | +| reason | `String` | An optional reason related to the status of the execution | `Cancelled by user`, `Failed testCase` | | +| testSuite | `Object` [`testSuite`](#testsuite) | An optional definition of the testSuite being executed | `{"id": "92834723894", "name": "Auth TestSuite"}` | | ### `testOutput published` @@ -182,7 +181,7 @@ The event represents a test execution output artifact that has been published. |-------------|----------------------------------------|---------------------------------------------------------------------------|-----------------------------------------------------------------------|----------| | id | `String` | Uniquely identifies the subject within the source. | `12312334` | ✅ | | source | `URI-Reference` | [source](spec.md#source--context-) from the context | | ✅ | -| type | `String` | The type of output, one of `report`, `video`, `image`, `log`, `other` | `video` | ✅ | +| outputType | `String` | The type of output, one of `report`, `video`, `image`, `log`, `other` | `video` | ✅ | | format | `String` | The Content-Type of the output artifact | `application/pdf`, `image/png`, `application/json` | ✅ | | uri | `URI-Reference` | A reference to retrieve the specified output artifact | `https://testkube.mycluster.internal/artifacts/23123123` | ✅ | | testCaseRun | `Object` [`testCaseRun`](#testcaserun) | An optional testCaseRun to link this artifact to a specific `testCaseRun` | `{"id":"Login-Test-execution-1213423", "source": "staging/testkube"}` | | @@ -200,7 +199,7 @@ A `trigger` in this context is what started a corresponding testCaseRun/testSuit ### `testCase` -A `testCase` is the actual test that is being run by a `testCaseRun`. +A `testCase` is the actual test that is being run by a `testCaseRun`. | Field | Type | Description | Examples | Required | |--------|-----------------|-----------------------------------------------------|----------------------------|----------| @@ -213,7 +212,7 @@ A `testCase` is the actual test that is being run by a `testCaseRun`. ### `testSuite` -A `testSuite` is a collection of `testCase` objects managed in an external system. Each time a `testSuite` is executed the +A `testSuite` is a collection of `testCase` objects managed in an external system. Each time a `testSuite` is executed the corresponding `testSuiteXXX` events should be emmitted. | Field | Type | Description | Examples | Required | From 9c86a9a3b3249e1f3c1869254cf93c91452c6179 Mon Sep 17 00:00:00 2001 From: olensmar Date: Thu, 30 Mar 2023 15:00:17 +0200 Subject: [PATCH 20/37] fixed linting and example errors --- examples/testcaserun_finished.json | 3 ++- examples/testcaserun_queued.json | 1 + examples/testcaserun_started.json | 1 + examples/testoutput_published.json | 2 +- examples/testsuiterun_finished.json | 1 + examples/testsuiterun_queued.json | 1 + examples/testsuiterun_started.json | 1 + schemas/testoutputpublished.json | 5 ++--- spec.md | 30 ++++++++++++++--------------- testing-events.md | 2 +- 10 files changed, 26 insertions(+), 21 deletions(-) diff --git a/examples/testcaserun_finished.json b/examples/testcaserun_finished.json index 29c0e3ba..fa903ab2 100644 --- a/examples/testcaserun_finished.json +++ b/examples/testcaserun_finished.json @@ -3,7 +3,7 @@ "version": "0.2.0", "id": "271069a8-fc18-44f1-b38f-9d70a1695819", "source": "/event/source/123", - "type": "dev.cdevents.testcaserun.started.0.1.0-draft", + "type": "dev.cdevents.testcaserun.finished.0.1.0-draft", "timestamp": "2023-03-20T14:27:05.315384Z" }, "subject": { @@ -18,6 +18,7 @@ }, "testCase": { "id": "92834723894", + "source": "testkube-dev-123", "name": "Login Test", "type": "integration" } diff --git a/examples/testcaserun_queued.json b/examples/testcaserun_queued.json index e6e311ee..6981e810 100644 --- a/examples/testcaserun_queued.json +++ b/examples/testcaserun_queued.json @@ -17,6 +17,7 @@ }, "testCase": { "id": "92834723894", + "source": "testkube-dev-123", "name": "Login Test", "type": "integration" }, diff --git a/examples/testcaserun_started.json b/examples/testcaserun_started.json index dc959827..a134ec9a 100644 --- a/examples/testcaserun_started.json +++ b/examples/testcaserun_started.json @@ -17,6 +17,7 @@ }, "testCase": { "id": "92834723894", + "source": "testkube-dev-123", "name": "Login Test", "type": "integration" }, diff --git a/examples/testoutput_published.json b/examples/testoutput_published.json index 23137740..9e54572e 100644 --- a/examples/testoutput_published.json +++ b/examples/testoutput_published.json @@ -15,7 +15,7 @@ "format" : "video/quicktime", "testCaseRun": { "id": "myTestCaseRun123", - "source": "/event/source/123" + "source": "testkube-dev-123" } } } diff --git a/examples/testsuiterun_finished.json b/examples/testsuiterun_finished.json index 7ba06aa6..a6274b70 100644 --- a/examples/testsuiterun_finished.json +++ b/examples/testsuiterun_finished.json @@ -20,6 +20,7 @@ }, "testSuite": { "id": "92834723894", + "source": "testkube-dev-123", "name": "Auth TestSuite" } } diff --git a/examples/testsuiterun_queued.json b/examples/testsuiterun_queued.json index bd377804..98200e5f 100644 --- a/examples/testsuiterun_queued.json +++ b/examples/testsuiterun_queued.json @@ -17,6 +17,7 @@ }, "testSuite": { "id": "92834723894", + "source": "testkube-dev-123", "name": "Auth TestSuite" }, "trigger": { diff --git a/examples/testsuiterun_started.json b/examples/testsuiterun_started.json index 8b96855e..18127b9c 100644 --- a/examples/testsuiterun_started.json +++ b/examples/testsuiterun_started.json @@ -17,6 +17,7 @@ }, "testSuite": { "id": "92834723894", + "source": "testkube-dev-123", "name": "Auth TestSuite" }, "trigger": { diff --git a/schemas/testoutputpublished.json b/schemas/testoutputpublished.json index 63e5cf4a..180e365a 100644 --- a/schemas/testoutputpublished.json +++ b/schemas/testoutputpublished.json @@ -94,9 +94,8 @@ "additionalProperties": false, "type": "object", "required": [ - "type", - "format", - "uri" + "outputType", + "format" ] } }, diff --git a/spec.md b/spec.md index b759f4b2..0b1565d1 100644 --- a/spec.md +++ b/spec.md @@ -27,19 +27,19 @@ CDEvents is a common specification for Continuous Delivery events. - [Types](#types) - [CDEvent context](#cdevent-context) - [REQUIRED Context Attributes](#required-context-attributes) - - [id (context)](#id--context-) - - [type (context)](#type--context-) - - [source (context)](#source--context-) + - [id (context)](#id-context) + - [type (context)](#type-context) + - [source (context)](#source-context) - [timestamp](#timestamp) - [version](#version) - [Context example](#context-example) - [CDEvent subject](#cdevent-subject) - [REQUIRED Subject Attributes](#required-subject-attributes) - - [id (subject)](#id--subject-) + - [id (subject)](#id-subject) - [content](#content) - [OPTIONAL Subject Attributes](#optional-subject-attributes) - - [source (subject)](#source--subject-) - - [type (subject)](#type--subject-) + - [source (subject)](#source-subject) + - [type (subject)](#type-subject) - [Subject example](#subject-example) - [CDEvents custom data](#cdevents-custom-data) - [OPTIONAL Custom Data attributes](#optional-custom-data-attributes) @@ -170,14 +170,14 @@ defined in the [vocabulary](#vocabulary): - Type: [`String`][typesystem] - Description: Identifier for an event. Subsequent delivery attempts of the same event MAY share the same - [`id`](#id--context-). This attribute matches the syntax and semantics of the + [`id`](#id-context). This attribute matches the syntax and semantics of the [`id`](https://github.com/cloudevents/spec/blob/v1.0.2/cloudevents/spec.md#id) attribute of CloudEvents. - Constraints: - REQUIRED - MUST be a non-empty string - - MUST be unique within the given [`source`](#source--context-) (in the scope of + - MUST be unique within the given [`source`](#source-context) (in the scope of the producer) - Examples: - A [UUID version 4](https://en.wikipedia.org/wiki/Universally_unique_identifier#Version_4_(random)) @@ -204,8 +204,8 @@ defined in the [vocabulary](#vocabulary): - Type: [`URI-Reference`][typesystem] - Description: defines the context in which an event happened. The main purpose - of the source is to provide global uniqueness for [`source`](#source--context-) + - [`id`](#id--context-). + of the source is to provide global uniqueness for [`source`](#source-context) + + [`id`](#id-context). The source MAY identify a single producer or a group of producer that belong to the same application. @@ -244,7 +244,7 @@ defined in the [vocabulary](#vocabulary): In case the transport layer should require a re-transmission of the event, the timestamp SHOULD NOT be updated, i.e. it should be the same for the same - [`source`](#source--context-) + [`id`](#id--context-) combination. + [`source`](#source-context) + [`id`](#id-context) combination. - Constraints: - REQUIRED @@ -305,7 +305,7 @@ defined in the [vocabulary](#vocabulary): - Type: [`Object`](#types) - Description: This provides all the relevant details of the [`content`](#content). The format of the [`content`](#content) depends on the - event [`type`](#type--context-). All attributes in the subject + event [`type`](#type-context). All attributes in the subject [`content`](#content), REQUIRED and OPTIONAL ones, MUST comply with the specification from the [vocabulary](#vocabulary). The [`content`](#content) may be empty. @@ -331,12 +331,12 @@ defined in the [vocabulary](#vocabulary): - Type: [`URI-Reference`][typesystem] - Description: defines the context in which the subject originated. In most cases the [`source`](#source--subject-) of the subject matches the - [`source`](#source--context-) of the event. This field should be used only in + [`source`](#source-context) of the event. This field should be used only in cases where the [`source`](#source--subject-) of the *subject* is different from - the [`source`](#source--context-) of the event. + the [`source`](#source-context) of the event. The format and semantic of the *subject* [`source`](#source--subject-) are the - same of those of the *context* [`source`](#source--context-). + same of those of the *context* [`source`](#source-context). #### type (subject) diff --git a/testing-events.md b/testing-events.md index 9df4344e..ae3fd5ed 100644 --- a/testing-events.md +++ b/testing-events.md @@ -183,7 +183,7 @@ The event represents a test execution output artifact that has been published. | source | `URI-Reference` | [source](spec.md#source--context-) from the context | | ✅ | | outputType | `String` | The type of output, one of `report`, `video`, `image`, `log`, `other` | `video` | ✅ | | format | `String` | The Content-Type of the output artifact | `application/pdf`, `image/png`, `application/json` | ✅ | -| uri | `URI-Reference` | A reference to retrieve the specified output artifact | `https://testkube.mycluster.internal/artifacts/23123123` | ✅ | +| uri | `URI-Reference` | A reference to retrieve the specified output artifact | `https://testkube.mycluster.internal/artifacts/23123123` | | | testCaseRun | `Object` [`testCaseRun`](#testcaserun) | An optional testCaseRun to link this artifact to a specific `testCaseRun` | `{"id":"Login-Test-execution-1213423", "source": "staging/testkube"}` | | ## Common Objects From 0779d09f6bf2863c688d9a0cfb7be14b41647324 Mon Sep 17 00:00:00 2001 From: olensmar Date: Thu, 30 Mar 2023 15:04:07 +0200 Subject: [PATCH 21/37] fixed linting and example errors --- schemas/testoutputpublished.json | 4 ++-- spec.md | 10 +++++----- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/schemas/testoutputpublished.json b/schemas/testoutputpublished.json index 180e365a..362a9f0b 100644 --- a/schemas/testoutputpublished.json +++ b/schemas/testoutputpublished.json @@ -53,8 +53,8 @@ }, "content": { "properties": { - "type": { - "outputType": "string", + "outputType": { + "type": "string", "enum": [ "report", "video", diff --git a/spec.md b/spec.md index 0b1565d1..b0f96e08 100644 --- a/spec.md +++ b/spec.md @@ -290,12 +290,12 @@ defined in the [vocabulary](#vocabulary): - Type: [`String`][typesystem] - Description: Identifier for a subject. Subsequent events associated to the same subject MUST use the same subject - [`id`](#id--subject-). + [`id`](#id-subject). - Constraints: - REQUIRED - MUST be a non-empty string - - MUST be unique within the given [`source`](#source--subject-) (in the scope of + - MUST be unique within the given [`source`](#source-subject) (in the scope of the producer) - Examples: - A [UUID version 4](https://en.wikipedia.org/wiki/Universally_unique_identifier#Version_4_(random)) @@ -330,12 +330,12 @@ defined in the [vocabulary](#vocabulary): - Type: [`URI-Reference`][typesystem] - Description: defines the context in which the subject originated. In most - cases the [`source`](#source--subject-) of the subject matches the + cases the [`source`](#source-subject) of the subject matches the [`source`](#source-context) of the event. This field should be used only in - cases where the [`source`](#source--subject-) of the *subject* is different from + cases where the [`source`](#source-subject) of the *subject* is different from the [`source`](#source-context) of the event. - The format and semantic of the *subject* [`source`](#source--subject-) are the + The format and semantic of the *subject* [`source`](#source-subject) are the same of those of the *context* [`source`](#source-context). #### type (subject) From 69ad13c6f115ab3baa9a1dcb3604398a64cdd3e6 Mon Sep 17 00:00:00 2001 From: olensmar Date: Thu, 30 Mar 2023 15:15:30 +0200 Subject: [PATCH 22/37] fixed linting and example errors --- examples/testcase_finished.json | 15 ---- examples/testcase_queued.json | 15 ---- examples/testcase_started.json | 15 ---- examples/testsuite_finished.json | 15 ---- examples/testsuite_started.json | 15 ---- examples/testsuiterun_finished.json | 2 +- schemas/testsuiterunfinished.json | 134 ++++++++++++++-------------- 7 files changed, 66 insertions(+), 145 deletions(-) delete mode 100644 examples/testcase_finished.json delete mode 100644 examples/testcase_queued.json delete mode 100644 examples/testcase_started.json delete mode 100644 examples/testsuite_finished.json delete mode 100644 examples/testsuite_started.json diff --git a/examples/testcase_finished.json b/examples/testcase_finished.json deleted file mode 100644 index 373b4694..00000000 --- a/examples/testcase_finished.json +++ /dev/null @@ -1,15 +0,0 @@ -{ - "context": { - "version": "0.2.0", - "id": "271069a8-fc18-44f1-b38f-9d70a1695819", - "source": "/event/source/123", - "type": "dev.cdevents.testcase.finished.0.1.1", - "timestamp": "2023-03-20T14:27:05.315384Z" - }, - "subject": { - "id": "mySubject123", - "source": "/event/source/123", - "type": "testCase", - "content": {} - } -} diff --git a/examples/testcase_queued.json b/examples/testcase_queued.json deleted file mode 100644 index b4bca40f..00000000 --- a/examples/testcase_queued.json +++ /dev/null @@ -1,15 +0,0 @@ -{ - "context": { - "version": "0.2.0", - "id": "271069a8-fc18-44f1-b38f-9d70a1695819", - "source": "/event/source/123", - "type": "dev.cdevents.testcase.queued.0.1.1", - "timestamp": "2023-03-20T14:27:05.315384Z" - }, - "subject": { - "id": "mySubject123", - "source": "/event/source/123", - "type": "testCase", - "content": {} - } -} diff --git a/examples/testcase_started.json b/examples/testcase_started.json deleted file mode 100644 index 2ffa1dc4..00000000 --- a/examples/testcase_started.json +++ /dev/null @@ -1,15 +0,0 @@ -{ - "context": { - "version": "0.2.0", - "id": "271069a8-fc18-44f1-b38f-9d70a1695819", - "source": "/event/source/123", - "type": "dev.cdevents.testcase.started.0.1.1", - "timestamp": "2023-03-20T14:27:05.315384Z" - }, - "subject": { - "id": "mySubject123", - "source": "/event/source/123", - "type": "testCase", - "content": {} - } -} diff --git a/examples/testsuite_finished.json b/examples/testsuite_finished.json deleted file mode 100644 index ac74f6a2..00000000 --- a/examples/testsuite_finished.json +++ /dev/null @@ -1,15 +0,0 @@ -{ - "context": { - "version": "0.2.0", - "id": "271069a8-fc18-44f1-b38f-9d70a1695819", - "source": "/event/source/123", - "type": "dev.cdevents.testsuite.finished.0.1.1", - "timestamp": "2023-03-20T14:27:05.315384Z" - }, - "subject": { - "id": "mySubject123", - "source": "/event/source/123", - "type": "testSuite", - "content": {} - } -} diff --git a/examples/testsuite_started.json b/examples/testsuite_started.json deleted file mode 100644 index 8cd8b920..00000000 --- a/examples/testsuite_started.json +++ /dev/null @@ -1,15 +0,0 @@ -{ - "context": { - "version": "0.2.0", - "id": "271069a8-fc18-44f1-b38f-9d70a1695819", - "source": "/event/source/123", - "type": "dev.cdevents.testsuite.started.0.1.1", - "timestamp": "2023-03-20T14:27:05.315384Z" - }, - "subject": { - "id": "mySubject123", - "source": "/event/source/123", - "type": "testSuite", - "content": {} - } -} diff --git a/examples/testsuiterun_finished.json b/examples/testsuiterun_finished.json index a6274b70..93126368 100644 --- a/examples/testsuiterun_finished.json +++ b/examples/testsuiterun_finished.json @@ -11,7 +11,7 @@ "source": "/event/source/123", "type": "testSuiteRun", "content": { - "status" : "failed", + "status" : "fail", "severity" : "critical", "reason" : "Host 123.34.23.32 not found", "environment": { diff --git a/schemas/testsuiterunfinished.json b/schemas/testsuiterunfinished.json index a6f35e49..c5985dea 100644 --- a/schemas/testsuiterunfinished.json +++ b/schemas/testsuiterunfinished.json @@ -53,82 +53,78 @@ }, "content": { "properties": { - "properties": { - "environment": { - "description": "the environment in which this testSuiteRun started", - "properties": { - "id": { - "type": "string", - "minLength": 1 - }, - "source": { - "type": "string" - }, - "name": { - "type": "string" - }, - "url": { - "type": "string", - "format": "uri" - } + "environment": { + "description": "the environment in which this testSuiteRun started", + "properties": { + "id": { + "type": "string", + "minLength": 1 }, - "additionalProperties": false, - "type": "object", - "required": [ - "id" - ] - }, - "testSuite": { - "description": "the testSuite which this testSuiteRun is executing", - "properties": { - "id": { - "type": "string", - "minLength": 1 - }, - "source": { - "type": "string" - }, - "name": { - "type": "string" - }, - "url": { - "type": "string", - "format": "uri" - } + "source": { + "type": "string" + }, + "name": { + "type": "string" + }, + "url": { + "type": "string", + "format": "uri" } }, - "status": { - "type": "string", - "enum": [ - "pass", - "fail", - "cancel", - "error" - ] - }, - "severity": { - "type": "string", - "enum": [ - "low", - "medium", - "high", - "critical" - ] - }, - "reason": { - "type": "string", - "description": "an optional reason to why the testSuite ended as it did" + "additionalProperties": false, + "type": "object", + "required": [ + "id" + ] + }, + "testSuite": { + "description": "the testSuite which this testSuiteRun is executing", + "properties": { + "id": { + "type": "string", + "minLength": 1 + }, + "source": { + "type": "string" + }, + "name": { + "type": "string" + }, + "url": { + "type": "string", + "format": "uri" + } } }, - "additionalProperties": false, - "type": "object", - "required": [ - "status", - "environment" - ] + "status": { + "type": "string", + "enum": [ + "pass", + "fail", + "cancel", + "error" + ] + }, + "severity": { + "type": "string", + "enum": [ + "low", + "medium", + "high", + "critical" + ] + }, + "reason": { + "type": "string", + "description": "an optional reason to why the testSuite ended as it did" + } }, "additionalProperties": false, - "type": "object" + "type": "object", + "required": [ + "status", + "environment" + ] } }, "additionalProperties": false, From 9d2f24ba05e9c33d537b1a1119240c93a33ebef9 Mon Sep 17 00:00:00 2001 From: olensmar Date: Thu, 30 Mar 2023 16:52:02 +0200 Subject: [PATCH 23/37] added missing enum and fixed type removal --- continuous-integration.md | 1 + schemas/testcaserunfinished.json | 6 +++++- schemas/testcaserunqueued.json | 6 +++++- schemas/testcaserunstarted.json | 6 +++++- schemas/testoutputpublished.json | 6 +++++- schemas/testsuiterunfinished.json | 6 +++++- schemas/testsuiterunqueued.json | 6 +++++- schemas/testsuiterunstarted.json | 6 +++++- 8 files changed, 36 insertions(+), 7 deletions(-) diff --git a/continuous-integration.md b/continuous-integration.md index d030a125..728652d0 100644 --- a/continuous-integration.md +++ b/continuous-integration.md @@ -33,6 +33,7 @@ __Note:__ The data model for `builds`, apart from `id` and `source`, only includ |-------|------|-------------|----------| | id | `String` | Uniquely identifies the subject within the source. | `1234`, `maven123`, `builds/taskrun123` | | source | `URI-Reference` | [source](../spec.md#source) from the context | `staging/tekton`, `tekton-dev-123`| +| type | `String` | See [type](spec.md#type-subject) | `build` | | artifactId | `String` | Identifier of the artifact produced by the build | `pkg:oci/myapp@sha256%3A0b31b1c02ff458ad9b7b81cbdf8f028bd54699fa151f221d1e8de6817db93427`, `pkg:golang/mygit.com/myorg/myapp@234fd47e07d1004f0aed9c` | ### `artifact` diff --git a/schemas/testcaserunfinished.json b/schemas/testcaserunfinished.json index 71c410a5..b975051b 100644 --- a/schemas/testcaserunfinished.json +++ b/schemas/testcaserunfinished.json @@ -49,7 +49,11 @@ }, "type": { "type": "string", - "minLength": 1 + "minLength": 1, + "enum": [ + "testCaseRun" + ], + "default": "testCaseRun" }, "content": { "properties": { diff --git a/schemas/testcaserunqueued.json b/schemas/testcaserunqueued.json index f836636f..ba32849a 100644 --- a/schemas/testcaserunqueued.json +++ b/schemas/testcaserunqueued.json @@ -49,7 +49,11 @@ }, "type": { "type": "string", - "minLength": 1 + "minLength": 1, + "enum": [ + "testCaseRun" + ], + "default": "testCaseRun" }, "content": { "properties": { diff --git a/schemas/testcaserunstarted.json b/schemas/testcaserunstarted.json index 91422b65..47040426 100644 --- a/schemas/testcaserunstarted.json +++ b/schemas/testcaserunstarted.json @@ -49,7 +49,11 @@ }, "type": { "type": "string", - "minLength": 1 + "minLength": 1, + "enum": [ + "testCaseRun" + ], + "default": "testCaseRun" }, "content": { "properties": { diff --git a/schemas/testoutputpublished.json b/schemas/testoutputpublished.json index 362a9f0b..2ccd3a26 100644 --- a/schemas/testoutputpublished.json +++ b/schemas/testoutputpublished.json @@ -49,7 +49,11 @@ }, "type": { "type": "string", - "minLength": 1 + "minLength": 1, + "enum": [ + "testOutput" + ], + "default": "testOutput" }, "content": { "properties": { diff --git a/schemas/testsuiterunfinished.json b/schemas/testsuiterunfinished.json index c5985dea..097aee0c 100644 --- a/schemas/testsuiterunfinished.json +++ b/schemas/testsuiterunfinished.json @@ -49,7 +49,11 @@ }, "type": { "type": "string", - "minLength": 1 + "minLength": 1, + "enum": [ + "testSuiteRun" + ], + "default": "testSuiteRun" }, "content": { "properties": { diff --git a/schemas/testsuiterunqueued.json b/schemas/testsuiterunqueued.json index ec635514..4c3e5ce7 100644 --- a/schemas/testsuiterunqueued.json +++ b/schemas/testsuiterunqueued.json @@ -49,7 +49,11 @@ }, "type": { "type": "string", - "minLength": 1 + "minLength": 1, + "enum": [ + "testSuiteRun" + ], + "default": "testSuiteRun" }, "content": { "properties": { diff --git a/schemas/testsuiterunstarted.json b/schemas/testsuiterunstarted.json index c6a62347..3c6c7dc7 100644 --- a/schemas/testsuiterunstarted.json +++ b/schemas/testsuiterunstarted.json @@ -49,7 +49,11 @@ }, "type": { "type": "string", - "minLength": 1 + "minLength": 1, + "enum": [ + "testSuiteRun" + ], + "default": "testSuiteRun" }, "content": { "properties": { From 7d1d09da046ea03c41c0577ba0562143383c4f59 Mon Sep 17 00:00:00 2001 From: olensmar Date: Thu, 30 Mar 2023 17:17:48 +0200 Subject: [PATCH 24/37] added missing object properties --- schemas/testcaserunfinished.json | 6 ++++++ schemas/testcaserunqueued.json | 6 ++++++ schemas/testsuiterunfinished.json | 6 ++++++ schemas/testsuiterunqueued.json | 6 ++++++ schemas/testsuiterunstarted.json | 6 ++++++ 5 files changed, 30 insertions(+) diff --git a/schemas/testcaserunfinished.json b/schemas/testcaserunfinished.json index b975051b..178de96f 100644 --- a/schemas/testcaserunfinished.json +++ b/schemas/testcaserunfinished.json @@ -122,6 +122,12 @@ }, "testCase": { "description": "the testCase which this testCaseRun is executing", + "type": "object", + "additionalProperties": false, + "required": [ + "id", + "source" + ], "properties": { "id": { "type": "string", diff --git a/schemas/testcaserunqueued.json b/schemas/testcaserunqueued.json index ba32849a..b0e6cac0 100644 --- a/schemas/testcaserunqueued.json +++ b/schemas/testcaserunqueued.json @@ -120,6 +120,12 @@ }, "testCase": { "description": "the testCase which this testCaseRun is executing", + "type": "object", + "additionalProperties": false, + "required": [ + "id", + "source" + ], "properties": { "id": { "type": "string", diff --git a/schemas/testsuiterunfinished.json b/schemas/testsuiterunfinished.json index 097aee0c..f7364515 100644 --- a/schemas/testsuiterunfinished.json +++ b/schemas/testsuiterunfinished.json @@ -83,6 +83,12 @@ }, "testSuite": { "description": "the testSuite which this testSuiteRun is executing", + "type": "object", + "additionalProperties": "false", + "required": [ + "id", + "source" + ], "properties": { "id": { "type": "string", diff --git a/schemas/testsuiterunqueued.json b/schemas/testsuiterunqueued.json index 4c3e5ce7..44bb12ca 100644 --- a/schemas/testsuiterunqueued.json +++ b/schemas/testsuiterunqueued.json @@ -103,6 +103,12 @@ }, "testSuite": { "description": "the testSuite which this testSuiteRun is executing", + "type": "object", + "additionalProperties": "false", + "required": [ + "id", + "source" + ], "properties": { "id": { "type": "string", diff --git a/schemas/testsuiterunstarted.json b/schemas/testsuiterunstarted.json index 3c6c7dc7..a2fae1b7 100644 --- a/schemas/testsuiterunstarted.json +++ b/schemas/testsuiterunstarted.json @@ -103,6 +103,12 @@ }, "testSuite": { "description": "the testSuite which this testSuiteRun is executing", + "type": "object", + "additionalProperties": "false", + "required": [ + "id", + "source" + ], "properties": { "id": { "type": "string", From 4a0111a7f8eb5d9c6a2b48bd119faea3bd06ee2d Mon Sep 17 00:00:00 2001 From: olensmar Date: Thu, 30 Mar 2023 17:20:59 +0200 Subject: [PATCH 25/37] fixed boolean.. --- schemas/testsuiterunfinished.json | 2 +- schemas/testsuiterunqueued.json | 2 +- schemas/testsuiterunstarted.json | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/schemas/testsuiterunfinished.json b/schemas/testsuiterunfinished.json index f7364515..8da8a496 100644 --- a/schemas/testsuiterunfinished.json +++ b/schemas/testsuiterunfinished.json @@ -84,7 +84,7 @@ "testSuite": { "description": "the testSuite which this testSuiteRun is executing", "type": "object", - "additionalProperties": "false", + "additionalProperties": false, "required": [ "id", "source" diff --git a/schemas/testsuiterunqueued.json b/schemas/testsuiterunqueued.json index 44bb12ca..b303fde6 100644 --- a/schemas/testsuiterunqueued.json +++ b/schemas/testsuiterunqueued.json @@ -104,7 +104,7 @@ "testSuite": { "description": "the testSuite which this testSuiteRun is executing", "type": "object", - "additionalProperties": "false", + "additionalProperties": false, "required": [ "id", "source" diff --git a/schemas/testsuiterunstarted.json b/schemas/testsuiterunstarted.json index a2fae1b7..0af9c668 100644 --- a/schemas/testsuiterunstarted.json +++ b/schemas/testsuiterunstarted.json @@ -104,7 +104,7 @@ "testSuite": { "description": "the testSuite which this testSuiteRun is executing", "type": "object", - "additionalProperties": "false", + "additionalProperties": false, "required": [ "id", "source" From 833aad2f473d7bf3aca6add95bf5aadc75e066f3 Mon Sep 17 00:00:00 2001 From: olensmar Date: Tue, 4 Apr 2023 09:43:48 +0200 Subject: [PATCH 26/37] more review comments.. --- examples/testcaserun_finished.json | 2 +- examples/testcaserun_queued.json | 2 +- examples/testcaserun_started.json | 2 +- examples/testsuiterun_finished.json | 2 +- examples/testsuiterun_queued.json | 2 +- examples/testsuiterun_started.json | 2 +- schemas/testcaserunfinished.json | 7 +- schemas/testcaserunqueued.json | 7 +- schemas/testcaserunstarted.json | 7 +- schemas/testsuiterunfinished.json | 7 +- schemas/testsuiterunqueued.json | 4 +- schemas/testsuiterunstarted.json | 4 +- testing-events.md | 138 ++++++++++++++-------------- 13 files changed, 91 insertions(+), 95 deletions(-) diff --git a/examples/testcaserun_finished.json b/examples/testcaserun_finished.json index fa903ab2..17158617 100644 --- a/examples/testcaserun_finished.json +++ b/examples/testcaserun_finished.json @@ -18,7 +18,7 @@ }, "testCase": { "id": "92834723894", - "source": "testkube-dev-123", + "version": "1.0", "name": "Login Test", "type": "integration" } diff --git a/examples/testcaserun_queued.json b/examples/testcaserun_queued.json index 6981e810..71d5c12d 100644 --- a/examples/testcaserun_queued.json +++ b/examples/testcaserun_queued.json @@ -17,7 +17,7 @@ }, "testCase": { "id": "92834723894", - "source": "testkube-dev-123", + "version": "1.0", "name": "Login Test", "type": "integration" }, diff --git a/examples/testcaserun_started.json b/examples/testcaserun_started.json index a134ec9a..25b736a2 100644 --- a/examples/testcaserun_started.json +++ b/examples/testcaserun_started.json @@ -17,7 +17,7 @@ }, "testCase": { "id": "92834723894", - "source": "testkube-dev-123", + "version": "1.0", "name": "Login Test", "type": "integration" }, diff --git a/examples/testsuiterun_finished.json b/examples/testsuiterun_finished.json index 93126368..0d41d42e 100644 --- a/examples/testsuiterun_finished.json +++ b/examples/testsuiterun_finished.json @@ -20,7 +20,7 @@ }, "testSuite": { "id": "92834723894", - "source": "testkube-dev-123", + "version": "1.0", "name": "Auth TestSuite" } } diff --git a/examples/testsuiterun_queued.json b/examples/testsuiterun_queued.json index 98200e5f..1b6eb1e9 100644 --- a/examples/testsuiterun_queued.json +++ b/examples/testsuiterun_queued.json @@ -17,7 +17,7 @@ }, "testSuite": { "id": "92834723894", - "source": "testkube-dev-123", + "version": "1.0", "name": "Auth TestSuite" }, "trigger": { diff --git a/examples/testsuiterun_started.json b/examples/testsuiterun_started.json index 18127b9c..357278a6 100644 --- a/examples/testsuiterun_started.json +++ b/examples/testsuiterun_started.json @@ -17,7 +17,7 @@ }, "testSuite": { "id": "92834723894", - "source": "testkube-dev-123", + "version": "1.0", "name": "Auth TestSuite" }, "trigger": { diff --git a/schemas/testcaserunfinished.json b/schemas/testcaserunfinished.json index 178de96f..4f398f91 100644 --- a/schemas/testcaserunfinished.json +++ b/schemas/testcaserunfinished.json @@ -125,15 +125,14 @@ "type": "object", "additionalProperties": false, "required": [ - "id", - "source" + "id" ], "properties": { "id": { "type": "string", "minLength": 1 }, - "source": { + "version": { "type": "string" }, "name": { @@ -152,7 +151,7 @@ "other" ] }, - "url": { + "uri": { "type": "string", "format": "uri" } diff --git a/schemas/testcaserunqueued.json b/schemas/testcaserunqueued.json index b0e6cac0..7670bdc1 100644 --- a/schemas/testcaserunqueued.json +++ b/schemas/testcaserunqueued.json @@ -123,15 +123,14 @@ "type": "object", "additionalProperties": false, "required": [ - "id", - "source" + "id" ], "properties": { "id": { "type": "string", "minLength": 1 }, - "source": { + "version": { "type": "string" }, "name": { @@ -150,7 +149,7 @@ "other" ] }, - "url": { + "uri": { "type": "string", "format": "uri" } diff --git a/schemas/testcaserunstarted.json b/schemas/testcaserunstarted.json index 47040426..b5db3a25 100644 --- a/schemas/testcaserunstarted.json +++ b/schemas/testcaserunstarted.json @@ -125,7 +125,7 @@ "type": "string", "minLength": 1 }, - "source": { + "version": { "type": "string" }, "name": { @@ -144,7 +144,7 @@ "other" ] }, - "url": { + "uri": { "type": "string", "format": "uri" } @@ -152,8 +152,7 @@ "additionalProperties": false, "type": "object", "required": [ - "id", - "source" + "id" ] } }, diff --git a/schemas/testsuiterunfinished.json b/schemas/testsuiterunfinished.json index 8da8a496..a3b495ca 100644 --- a/schemas/testsuiterunfinished.json +++ b/schemas/testsuiterunfinished.json @@ -86,21 +86,20 @@ "type": "object", "additionalProperties": false, "required": [ - "id", - "source" + "id" ], "properties": { "id": { "type": "string", "minLength": 1 }, - "source": { + "version": { "type": "string" }, "name": { "type": "string" }, - "url": { + "uri": { "type": "string", "format": "uri" } diff --git a/schemas/testsuiterunqueued.json b/schemas/testsuiterunqueued.json index b303fde6..957a20c8 100644 --- a/schemas/testsuiterunqueued.json +++ b/schemas/testsuiterunqueued.json @@ -84,13 +84,13 @@ "type": "string", "minLength": 1 }, - "source": { + "version": { "type": "string" }, "name": { "type": "string" }, - "url": { + "uri": { "type": "string", "format": "uri" } diff --git a/schemas/testsuiterunstarted.json b/schemas/testsuiterunstarted.json index 0af9c668..099df843 100644 --- a/schemas/testsuiterunstarted.json +++ b/schemas/testsuiterunstarted.json @@ -84,13 +84,13 @@ "type": "string", "minLength": 1 }, - "source": { + "version": { "type": "string" }, "name": { "type": "string" }, - "url": { + "uri": { "type": "string", "format": "uri" } diff --git a/testing-events.md b/testing-events.md index ae3fd5ed..4559aa3d 100644 --- a/testing-events.md +++ b/testing-events.md @@ -16,11 +16,11 @@ Testing events covers the subjects and predicates related to test-execution perf ## Subjects -This specification defines three subjects in this stage: `testCase`, `testSuite` and `testOutput`. +This specification defines three subjects in this stage: `testCaseRun`, `testSuiteRun` and `testOutput`. | Subject | Description | Predicates | |---------------------------------|----------------------------------------------|------------------------------------------------------------------------------------------------------------| -| [`testCaseRun`](#testcaserun) | The execution of a software testCase | [`queued`](#testcaserun-queued), [`started`](#testcaserun-started) [`finished`](#testcaserun-finished) | +| [`testCaseRun`](#testcaserun) | The execution of a software testCase | [`queued`](#testcaserun-queued), [`started`](#testcaserun-started), [`finished`](#testcaserun-finished) | | [`testSuiteRun`](#testsuiterun) | The execution of a software testSuite | [`queued`](#testsuiterun-queued), [`started`](#testsuiterun-started), [`finished`](#testsuiterun-finished) | | [`testOutput`](#testoutput) | An output artifact produced by a testCaseRun | [`published`](#testoutput-published) | @@ -35,7 +35,7 @@ A `testCaseRun` is the smallest unit of testing that the user wants to track, `t | source | `URI-Reference` | [source](spec.md#source--context-) from the context | `staging/testkube`, `testkube-dev-123` | | type | `String` | An optional type of test | `functional`, `unit`, `performance`, `security` | | environment | `Object` [`environment`](continuous-deployment.md/#environment) | The environment in which this testCaseRun is executing | `{"id": "1234"}`, `{"id": "dev", "source": "testkube-dev-123"}` | -| testCase | `Object` [`testCase`](#testcase) | An optional definition of the testCase being executed | `{"id": "92834723894", "name": "Login Test", "type":"integration"}` | | +| testCase | `Object` [`testCase`](#testcase) | An optional definition of the testCase being executed | `{"id": "92834723894", "name": "Login Test", "type": "integration"}` | | | testSuiteRun | `Object` [`testSuiteRun`](#testsuiterun) | An optional testSuiteRun to associate this testCaseRun with a containing testSuiteRun | `{"id":"Auth-TestSuite-execution-12334", "source": "staging/testkube"}` | | ### `testSuiteRun` @@ -73,14 +73,14 @@ This event represents when a testCaseRun has been queued for execution - and is - Predicate: queued - Subject: [`testCaseRun`](#testcaserun) -| Field | Type | Description | Examples | Required | -|--------------|-----------------------------------------------------------------|---------------------------------------------------------------------------------------|-------------------------------------------------------------------------|----------| -| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | -| environment | `Object` [`environment`](continuous-deployment.md/#environment) | The environment in which this testCaseRun is queued | `{"id": "1234"}`, `{"id": "dev", "source": "testkube-dev-123"}` | ✅ | -| testCase | `Object` [`testCase`](#testcase) | An optional definition of the testCase being executed | `{"id": "92834723894", "name": "Login Test", "type":"integration"}` | | -| testSuiteRun | `Object` [`testSuiteRun`](#testsuiterun) | An optional testSuiteRun to associate this testCaseRun with a containing testSuiteRun | `{"id":"Auth-TestSuite-execution-12334", "source": "staging/testkube"}` | | -| trigger | `Object` [trigger](#trigger) | What triggered this testSuiteRun | `{"type": "schedule"}` | +| Field | Type | Description | Examples | Required | +|--------------|-----------------------------------------------------------------|-----------------------------------------------------------------------------|-------------------------------------------------------------------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| environment | `Object` [`environment`](continuous-deployment.md/#environment) | The environment in which this testCaseRun is queued | `{"id": "1234"}`, `{"id": "dev", "source": "testkube-dev-123"}` | ✅ | +| testCase | `Object` [`testCase`](#testcase) | Definition of the testCase being executed | `{"id": "92834723894", "name": "Login Test", "type": "integration"}` | | +| testSuiteRun | `Object` [`testSuiteRun`](#testsuiterun) | A testSuiteRun to associate this testCaseRun with a containing testSuiteRun | `{"id":"Auth-TestSuite-execution-12334", "source": "staging/testkube"}` | | +| trigger | `Object` [trigger](#trigger) | What triggered this testSuiteRun | `{"type": "schedule"}` | ### `testCaseRun started` @@ -90,14 +90,14 @@ This event represents a started testCase execution. - Predicate: started - Subject: [`testCaseRun`](#testcaserun) -| Field | Type | Description | Examples | Required | -|--------------|-----------------------------------------------------------------|---------------------------------------------------------------------------------------|-------------------------------------------------------------------------|----------| -| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | -| environment | `Object` [`environment`](continuous-deployment.md/#environment) | The environment in which this testCaseRun is running | `{"id": "1234"}`, `{"id": "dev", "source": "testkube-dev-123"}` | ✅ | -| testCase | `Object` [`testCase`](#testcase) | An optional definition of the testCase being executed | `{"id": "92834723894", "name": "Login Test", "type":"integration"}` | | -| testSuiteRun | `Object` [`testSuiteRun`](#testsuiterun) | An optional testSuiteRun to associate this testCaseRun with a containing testSuiteRun | `{"id":"Auth-TestSuite-execution-12334", "source": "staging/testkube"}` | | -| trigger | `Object` [trigger](#trigger) | What triggered this testSuiteRun | `{"type": "event"}` | +| Field | Type | Description | Examples | Required | +|--------------|-----------------------------------------------------------------|-----------------------------------------------------------------------------|-------------------------------------------------------------------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| environment | `Object` [`environment`](continuous-deployment.md/#environment) | The environment in which this testCaseRun is running | `{"id": "1234"}`, `{"id": "dev", "source": "testkube-dev-123"}` | ✅ | +| testCase | `Object` [`testCase`](#testcase) | Definition of the testCase being executed | `{"id": "92834723894", "name": "Login Test", "type": "integration"}` | | +| testSuiteRun | `Object` [`testSuiteRun`](#testsuiterun) | A testSuiteRun to associate this testCaseRun with a containing testSuiteRun | `{"id":"Auth-TestSuite-execution-12334", "source": "staging/testkube"}` | | +| trigger | `Object` [trigger](#trigger) | What triggered this testSuiteRun | `{"type": "event"}` | ### `testCaseRun finished` @@ -107,16 +107,16 @@ This event represents a finished testCase execution. The event will contain the - Predicate: finished - Subject: [`testCaseRun`](#testcaserun) -| Field | Type | Description | Examples | Required | -|--------------|-----------------------------------------------------------------|---------------------------------------------------------------------------------------|-------------------------------------------------------------------------|----------| -| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | -| environment | `Object` [`environment`](continuous-deployment.md/#environment) | The environment in which this testCaseRun was running | `{"id": "1234"}`, `{"id": "dev", "source": "testkube-dev-123"}` | ✅ | -| testCase | `Object` [`testCase`](#testcase) | An optional definition of the testCase being executed | `{"id": "92834723894", "name": "Login Test", "type":"integration"}` | | -| testSuiteRun | `Object` [`testSuiteRun`](#testsuiterun) | An optional testSuiteRun to associate this testCaseRun with a containing testSuiteRun | `{"id":"Auth-TestSuite-execution-12334", "source": "staging/testkube"}` | | -| status | `String` | The status of the testSuite execution, one of `pass`, `fail`, `cancel`, `error` | `pass` | ✅ | -| severity | `String` | An optional severity if the test failed, one of `low`, `medium`, `high`, `critical` | `critical` | -| reason | `String` | An optional reason related to the `status` of the execution | `Cancelled by user`, `Failed assertion`, `Timed out` | | +| Field | Type | Description | Examples | Required | +|--------------|-----------------------------------------------------------------|---------------------------------------------------------------------------------|-------------------------------------------------------------------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| environment | `Object` [`environment`](continuous-deployment.md/#environment) | The environment in which this testCaseRun was running | `{"id": "1234"}`, `{"id": "dev", "source": "testkube-dev-123"}` | ✅ | +| testCase | `Object` [`testCase`](#testcase) | Definition of the testCase being executed | `{"id": "92834723894", "name": "Login Test", "type": "integration"}` | | +| testSuiteRun | `Object` [`testSuiteRun`](#testsuiterun) | A testSuiteRun to associate this testCaseRun with a containing testSuiteRun | `{"id":"Auth-TestSuite-execution-12334", "source": "staging/testkube"}` | | +| status | `String` | The status of the testSuite execution, one of `pass`, `fail`, `cancel`, `error` | `pass` | ✅ | +| severity | `String` | Severity if the test failed, one of `low`, `medium`, `high`, `critical` | `critical` | +| reason | `String` | A reason related to the `status` of the execution | `Cancelled by user`, `Failed assertion`, `Timed out` | | ### `testSuiteRun queued` @@ -127,13 +127,13 @@ This event represents when a testSuiteRun has been queued for execution - and is - Predicate: queued - Subject: [`testSuiteRun`](#testsuiterun) -| Field | Type | Description | Examples | Required | -|-------------|-----------------------------------------------------------------|--------------------------------------------------------|-----------------------------------------------------------------|----------| -| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | -| environment | `Object` [`environment`](continuous-deployment.md/#environment) | The environment in which this testSuiteRun is queued | `{"id": "1234"}`, `{"id": "dev", "source": "testkube-dev-123"}` | ✅ | -| testSuite | `Object` [`testSuite`](#testsuite) | An optional definition of the testSuite being executed | `{"id": "92834723894", "name": "Auth TestSuite"}` | | -| trigger | `Object` [trigger](#trigger) | What triggered this testSuiteRun | `{"type": "manual"}` | +| Field | Type | Description | Examples | Required | +|-------------|-----------------------------------------------------------------|------------------------------------------------------|-----------------------------------------------------------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| environment | `Object` [`environment`](continuous-deployment.md/#environment) | The environment in which this testSuiteRun is queued | `{"id": "1234"}`, `{"id": "dev", "source": "testkube-dev-123"}` | ✅ | +| testSuite | `Object` [`testSuite`](#testsuite) | Definition of the testSuite being executed | `{"id": "92834723894", "name": "Auth TestSuite"}` | | +| trigger | `Object` [trigger](#trigger) | What triggered this testSuiteRun | `{"type": "manual"}` | ### `testSuiteRun started` @@ -143,13 +143,13 @@ This event represents a started testSuite execution. - Predicate: started - Subject: [`testSuiteRun`](#testsuiterun) -| Field | Type | Description | Examples | Required | -|-------------|-----------------------------------------------------------------|--------------------------------------------------------|-----------------------------------------------------------------|----------| -| id | `String` | Uniquely identifies the subject within the source. | `unit`, `e2e`, `security` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | -| environment | `Object` [`environment`](continuous-deployment.md/#environment) | The environment in which this testSuiteRun is running | `{"id": "1234"}`, `{"id": "dev", "source": "testkube-dev-123"}` | ✅ | -| testSuite | `Object` [`testSuite`](#testsuite) | An optional definition of the testSuite being executed | `{"id": "92834723894", "name": "Auth TestSuite"}` | | -| trigger | `Object` [trigger](#trigger) | What triggered this testSuiteRun | `{"type": "pipeline"}` | +| Field | Type | Description | Examples | Required | +|-------------|-----------------------------------------------------------------|-------------------------------------------------------|-----------------------------------------------------------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `unit`, `e2e`, `security` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| environment | `Object` [`environment`](continuous-deployment.md/#environment) | The environment in which this testSuiteRun is running | `{"id": "1234"}`, `{"id": "dev", "source": "testkube-dev-123"}` | ✅ | +| testSuite | `Object` [`testSuite`](#testsuite) | Definition of the testSuite being executed | `{"id": "92834723894", "name": "Auth TestSuite"}` | | +| trigger | `Object` [trigger](#trigger) | What triggered this testSuiteRun | `{"type": "pipeline"}` | ### `testSuiteRun finished` @@ -159,15 +159,15 @@ This event represents a finished testSuite execution. The event will contain the - Predicate: finished - Subject: [`testSuiteRun`](#testsuiterun) -| Field | Type | Description | Examples | Required | -|-------------|-----------------------------------------------------------------|-------------------------------------------------------------------------------------|-----------------------------------------------------------------|----------| -| id | `String` | Uniquely identifies the subject within the source. | `unit`, `e2e`, `security` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | -| environment | `Object` [`environment`](continuous-deployment.md/#environment) | The environment in which this testSuiteRun was running | `{"id": "1234"}`, `{"id": "dev", "source": "testkube-dev-123"}` | ✅ | -| status | `String` | The status of the testSuite execution, one of `pass`, `fail`, `cancel`, `error` | `fail` | ✅ | -| severity | `String` | An optional severity if the test failed, one of `low`, `medium`, `high`, `critical` | `critical`, `low`, `medium`, `high` | -| reason | `String` | An optional reason related to the status of the execution | `Cancelled by user`, `Failed testCase` | | -| testSuite | `Object` [`testSuite`](#testsuite) | An optional definition of the testSuite being executed | `{"id": "92834723894", "name": "Auth TestSuite"}` | | +| Field | Type | Description | Examples | Required | +|-------------|-----------------------------------------------------------------|---------------------------------------------------------------------------------|-----------------------------------------------------------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `unit`, `e2e`, `security` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| environment | `Object` [`environment`](continuous-deployment.md/#environment) | The environment in which this testSuiteRun was running | `{"id": "1234"}`, `{"id": "dev", "source": "testkube-dev-123"}` | ✅ | +| status | `String` | The status of the testSuite execution, one of `pass`, `fail`, `cancel`, `error` | `fail` | ✅ | +| severity | `String` | Severity if the test failed, one of `low`, `medium`, `high`, `critical` | `critical`, `low`, `medium`, `high` | +| reason | `String` | A reason related to the status of the execution | `Cancelled by user`, `Failed testCase` | | +| testSuite | `Object` [`testSuite`](#testsuite) | Definition of the testSuite being executed | `{"id": "92834723894", "name": "Auth TestSuite"}` | | ### `testOutput published` @@ -192,22 +192,22 @@ The event represents a test execution output artifact that has been published. A `trigger` in this context is what started a corresponding testCaseRun/testSuiteRun. -| Field | Type | Description | Examples | Required | -|-------------|-----------------|--------------------------------------------------------------------------------|----------|----------| -| type | `String` | The type of trigger, one of `manual`, `pipeline`, `event`, `schedule`, `other` | | ✅ | -| uri | `URI-Reference` | An optional uri reference to this trigger | | | +| Field | Type | Description | Examples | Required | +|-------|-----------------|--------------------------------------------------------------------------------|----------|----------| +| type | `String` | The type of trigger, one of `manual`, `pipeline`, `event`, `schedule`, `other` | | ✅ | +| uri | `URI-Reference` | A uri reference to this trigger | | | ### `testCase` A `testCase` is the actual test that is being run by a `testCaseRun`. -| Field | Type | Description | Examples | Required | -|--------|-----------------|-----------------------------------------------------|----------------------------|----------| -| id | `String` | Uniquely identifies the testCase within the source. | `12312334` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | ✅ | -| type | `String` | An optional type of test | `functional`,`performance` | | -| name | `String` | An user-friendly name for this testCase | `Login Test` | | -| uri | `URI-Reference` | An optional uri reference to this testCase | | | +| Field | Type | Description | Examples | Required | +|---------|-----------------|-------------------------------------------------------------------|----------------------------|----------| +| id | `String` | Uniquely identifies the testCase within a test management system. | `12312334` | ✅ | +| type | `String` | The type of test | `functional`,`performance` | | +| name | `String` | A user-friendly name for this testCase | `Login Test` | | +| version | `String` | The version of the testCase | `1.0` | | +| uri | `URI-Reference` | A uri reference to this testCase | | | ### `testSuite` @@ -215,10 +215,10 @@ A `testCase` is the actual test that is being run by a `testCaseRun`. A `testSuite` is a collection of `testCase` objects managed in an external system. Each time a `testSuite` is executed the corresponding `testSuiteXXX` events should be emmitted. -| Field | Type | Description | Examples | Required | -|--------|-----------------|------------------------------------------------------|------------------|----------| -| id | `String` | Uniquely identifies the testSuite within the source. | `12312334` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | ✅ | -| name | `String` | An user-friendly name for this testSuite | `Auth TestSuite` | | -| uri | `URI-Reference` | An optional uri reference to this testSuite | | | +| Field | Type | Description | Examples | Required | +|---------|-----------------|--------------------------------------------------------------------|------------------|----------| +| id | `String` | Uniquely identifies the testSuite within a test management system. | `12312334` | ✅ | +| name | `String` | A user-friendly name for this testSuite | `Auth TestSuite` | | +| version | `String` | The version of the testSuite | `1.0` | | +| uri | `URI-Reference` | A uri reference to this testSuite | | | From f6ec1b9fce2f0285fda6ee483373f0b00740732c Mon Sep 17 00:00:00 2001 From: olensmar Date: Tue, 4 Apr 2023 09:54:10 +0200 Subject: [PATCH 27/37] fixing stupid mistakes.. --- schemas/testsuiterunqueued.json | 9 ++++----- schemas/testsuiterunstarted.json | 14 +++++++------- 2 files changed, 11 insertions(+), 12 deletions(-) diff --git a/schemas/testsuiterunqueued.json b/schemas/testsuiterunqueued.json index 957a20c8..ed7f92d9 100644 --- a/schemas/testsuiterunqueued.json +++ b/schemas/testsuiterunqueued.json @@ -84,13 +84,13 @@ "type": "string", "minLength": 1 }, - "version": { + "source": { "type": "string" }, "name": { "type": "string" }, - "uri": { + "url": { "type": "string", "format": "uri" } @@ -106,15 +106,14 @@ "type": "object", "additionalProperties": false, "required": [ - "id", - "source" + "id" ], "properties": { "id": { "type": "string", "minLength": 1 }, - "source": { + "version": { "type": "string" }, "name": { diff --git a/schemas/testsuiterunstarted.json b/schemas/testsuiterunstarted.json index 099df843..693d83d4 100644 --- a/schemas/testsuiterunstarted.json +++ b/schemas/testsuiterunstarted.json @@ -84,13 +84,13 @@ "type": "string", "minLength": 1 }, - "version": { + "source": { "type": "string" }, "name": { "type": "string" }, - "uri": { + "url": { "type": "string", "format": "uri" } @@ -98,7 +98,8 @@ "additionalProperties": false, "type": "object", "required": [ - "id" + "id", + "source" ] }, "testSuite": { @@ -106,21 +107,20 @@ "type": "object", "additionalProperties": false, "required": [ - "id", - "source" + "id" ], "properties": { "id": { "type": "string", "minLength": 1 }, - "source": { + "version": { "type": "string" }, "name": { "type": "string" }, - "url": { + "uri": { "type": "string", "format": "uri" } From e9d42bd294cafd184f16f1359197dd4b99886646 Mon Sep 17 00:00:00 2001 From: olensmar Date: Fri, 14 Apr 2023 13:04:03 +0200 Subject: [PATCH 28/37] removed descriptions --- schemas/testcaserunfinished.json | 6 +----- schemas/testcaserunqueued.json | 4 ---- schemas/testcaserunstarted.json | 4 ---- schemas/testoutputpublished.json | 5 +---- schemas/testsuiterunfinished.json | 5 +---- schemas/testsuiterunqueued.json | 3 --- schemas/testsuiterunstarted.json | 3 --- 7 files changed, 3 insertions(+), 27 deletions(-) diff --git a/schemas/testcaserunfinished.json b/schemas/testcaserunfinished.json index 4f398f91..b2cf86db 100644 --- a/schemas/testcaserunfinished.json +++ b/schemas/testcaserunfinished.json @@ -76,11 +76,9 @@ ] }, "reason": { - "type": "string", - "description": "an optional reason to why the testCase ended as it did" + "type": "string" }, "environment": { - "description": "the environment in which this testCaseRun is being queued", "properties": { "id": { "type": "string", @@ -105,7 +103,6 @@ }, "testSuiteRun": { "type": "object", - "description": "an optional reference to the containing testSuiteRun", "properties": { "id": { "type": "string", @@ -121,7 +118,6 @@ ] }, "testCase": { - "description": "the testCase which this testCaseRun is executing", "type": "object", "additionalProperties": false, "required": [ diff --git a/schemas/testcaserunqueued.json b/schemas/testcaserunqueued.json index 7670bdc1..7fe4afa2 100644 --- a/schemas/testcaserunqueued.json +++ b/schemas/testcaserunqueued.json @@ -59,7 +59,6 @@ "properties": { "trigger": { "type": "object", - "description": "an optional definition of what triggered this testCase queueing", "properties": { "type": { "type": "string", @@ -78,7 +77,6 @@ } }, "environment": { - "description": "the environment in which this testCaseRun is being queued", "properties": { "id": { "type": "string", @@ -102,7 +100,6 @@ ] }, "testSuiteRun": { - "description": "an optional reference to the containing testSuiteRun", "properties": { "id": { "type": "string", @@ -119,7 +116,6 @@ ] }, "testCase": { - "description": "the testCase which this testCaseRun is executing", "type": "object", "additionalProperties": false, "required": [ diff --git a/schemas/testcaserunstarted.json b/schemas/testcaserunstarted.json index b5db3a25..4cf86c3e 100644 --- a/schemas/testcaserunstarted.json +++ b/schemas/testcaserunstarted.json @@ -59,7 +59,6 @@ "properties": { "trigger": { "type": "object", - "description": "an optional definition of what triggered this testCase execution", "properties": { "type": { "type": "string", @@ -78,7 +77,6 @@ } }, "environment": { - "description": "the environment in which this testCaseRun started", "properties": { "id": { "type": "string", @@ -103,7 +101,6 @@ }, "testSuiteRun": { "type": "object", - "description": "an optional reference to the containing testSuiteRun", "properties": { "id": { "type": "string", @@ -119,7 +116,6 @@ ] }, "testCase": { - "description": "the testCase which this testCaseRun is executing", "properties": { "id": { "type": "string", diff --git a/schemas/testoutputpublished.json b/schemas/testoutputpublished.json index 2ccd3a26..f2959220 100644 --- a/schemas/testoutputpublished.json +++ b/schemas/testoutputpublished.json @@ -69,17 +69,14 @@ }, "format": { "type": "string", - "description": "The content-type of the output artifact", "example": "application/pdf" }, "uri": { "type": "string", - "format": "uri", - "description": "an optional URI reference to the actual testOutput artifact" + "format": "uri" }, "testCaseRun": { "type": "object", - "description": "an optional reference to the testCaseRun that generated this output", "properties": { "id": { "type": "string", diff --git a/schemas/testsuiterunfinished.json b/schemas/testsuiterunfinished.json index a3b495ca..e0c3eb8e 100644 --- a/schemas/testsuiterunfinished.json +++ b/schemas/testsuiterunfinished.json @@ -58,7 +58,6 @@ "content": { "properties": { "environment": { - "description": "the environment in which this testSuiteRun started", "properties": { "id": { "type": "string", @@ -82,7 +81,6 @@ ] }, "testSuite": { - "description": "the testSuite which this testSuiteRun is executing", "type": "object", "additionalProperties": false, "required": [ @@ -124,8 +122,7 @@ ] }, "reason": { - "type": "string", - "description": "an optional reason to why the testSuite ended as it did" + "type": "string" } }, "additionalProperties": false, diff --git a/schemas/testsuiterunqueued.json b/schemas/testsuiterunqueued.json index ed7f92d9..a9ae95d2 100644 --- a/schemas/testsuiterunqueued.json +++ b/schemas/testsuiterunqueued.json @@ -59,7 +59,6 @@ "properties": { "trigger": { "type": "object", - "description": "an optional definition of what triggered this testCase execution", "properties": { "type": { "type": "string", @@ -78,7 +77,6 @@ } }, "environment": { - "description": "the environment in which this testSuiteRun started", "properties": { "id": { "type": "string", @@ -102,7 +100,6 @@ ] }, "testSuite": { - "description": "the testSuite which this testSuiteRun is executing", "type": "object", "additionalProperties": false, "required": [ diff --git a/schemas/testsuiterunstarted.json b/schemas/testsuiterunstarted.json index 693d83d4..da8d55fb 100644 --- a/schemas/testsuiterunstarted.json +++ b/schemas/testsuiterunstarted.json @@ -59,7 +59,6 @@ "properties": { "trigger": { "type": "object", - "description": "an optional definition of what triggered this testCase execution", "properties": { "type": { "type": "string", @@ -78,7 +77,6 @@ } }, "environment": { - "description": "the environment in which this testSuiteRun started", "properties": { "id": { "type": "string", @@ -103,7 +101,6 @@ ] }, "testSuite": { - "description": "the testSuite which this testSuiteRun is executing", "type": "object", "additionalProperties": false, "required": [ From ffa8b122be18b1e92e2f166293e127be72280552 Mon Sep 17 00:00:00 2001 From: olensmar Date: Fri, 14 Apr 2023 13:13:15 +0200 Subject: [PATCH 29/37] updated deployment object and spec for type to include enum values --- schemas/testcaserunfinished.json | 9 ++------- schemas/testcaserunqueued.json | 9 ++------- schemas/testcaserunstarted.json | 9 ++------- schemas/testsuiterunfinished.json | 9 ++------- schemas/testsuiterunqueued.json | 9 ++------- schemas/testsuiterunstarted.json | 12 +++--------- testing-events.md | 14 +++++++------- 7 files changed, 20 insertions(+), 51 deletions(-) diff --git a/schemas/testcaserunfinished.json b/schemas/testcaserunfinished.json index b2cf86db..b35d07dc 100644 --- a/schemas/testcaserunfinished.json +++ b/schemas/testcaserunfinished.json @@ -85,14 +85,9 @@ "minLength": 1 }, "source": { - "type": "string" - }, - "name": { - "type": "string" - }, - "url": { "type": "string", - "format": "uri" + "minLength": 1, + "format": "uri-reference" } }, "additionalProperties": false, diff --git a/schemas/testcaserunqueued.json b/schemas/testcaserunqueued.json index 7fe4afa2..ec6f0746 100644 --- a/schemas/testcaserunqueued.json +++ b/schemas/testcaserunqueued.json @@ -83,14 +83,9 @@ "minLength": 1 }, "source": { - "type": "string" - }, - "name": { - "type": "string" - }, - "url": { "type": "string", - "format": "uri" + "minLength": 1, + "format": "uri-reference" } }, "additionalProperties": false, diff --git a/schemas/testcaserunstarted.json b/schemas/testcaserunstarted.json index 4cf86c3e..538bcd09 100644 --- a/schemas/testcaserunstarted.json +++ b/schemas/testcaserunstarted.json @@ -83,14 +83,9 @@ "minLength": 1 }, "source": { - "type": "string" - }, - "name": { - "type": "string" - }, - "url": { "type": "string", - "format": "uri" + "minLength": 1, + "format": "uri-reference" } }, "additionalProperties": false, diff --git a/schemas/testsuiterunfinished.json b/schemas/testsuiterunfinished.json index e0c3eb8e..9afe47c3 100644 --- a/schemas/testsuiterunfinished.json +++ b/schemas/testsuiterunfinished.json @@ -64,14 +64,9 @@ "minLength": 1 }, "source": { - "type": "string" - }, - "name": { - "type": "string" - }, - "url": { "type": "string", - "format": "uri" + "minLength": 1, + "format": "uri-reference" } }, "additionalProperties": false, diff --git a/schemas/testsuiterunqueued.json b/schemas/testsuiterunqueued.json index a9ae95d2..be4323ce 100644 --- a/schemas/testsuiterunqueued.json +++ b/schemas/testsuiterunqueued.json @@ -83,14 +83,9 @@ "minLength": 1 }, "source": { - "type": "string" - }, - "name": { - "type": "string" - }, - "url": { "type": "string", - "format": "uri" + "minLength": 1, + "format": "uri-reference" } }, "additionalProperties": false, diff --git a/schemas/testsuiterunstarted.json b/schemas/testsuiterunstarted.json index da8d55fb..a8cc3835 100644 --- a/schemas/testsuiterunstarted.json +++ b/schemas/testsuiterunstarted.json @@ -83,21 +83,15 @@ "minLength": 1 }, "source": { - "type": "string" - }, - "name": { - "type": "string" - }, - "url": { "type": "string", - "format": "uri" + "minLength": 1, + "format": "uri-reference" } }, "additionalProperties": false, "type": "object", "required": [ - "id", - "source" + "id" ] }, "testSuite": { diff --git a/testing-events.md b/testing-events.md index 4559aa3d..b2b1193e 100644 --- a/testing-events.md +++ b/testing-events.md @@ -201,13 +201,13 @@ A `trigger` in this context is what started a corresponding testCaseRun/testSuit A `testCase` is the actual test that is being run by a `testCaseRun`. -| Field | Type | Description | Examples | Required | -|---------|-----------------|-------------------------------------------------------------------|----------------------------|----------| -| id | `String` | Uniquely identifies the testCase within a test management system. | `12312334` | ✅ | -| type | `String` | The type of test | `functional`,`performance` | | -| name | `String` | A user-friendly name for this testCase | `Login Test` | | -| version | `String` | The version of the testCase | `1.0` | | -| uri | `URI-Reference` | A uri reference to this testCase | | | +| Field | Type | Description | Examples | Required | +|---------|-----------------|-----------------------------------------------------------------------------------------------------------------------|----------------------------|----------| +| id | `String` | Uniquely identifies the testCase within a test management system. | `12312334` | ✅ | +| type | `String enum` | The type of test, one of `performance`, `functional`, `unit`, `security`, `compliance`, `integration`, `e2e`, `other` | `functional`,`performance` | | +| name | `String` | A user-friendly name for this testCase | `Login Test` | | +| version | `String` | The version of the testCase | `1.0` | | +| uri | `URI-Reference` | A uri reference to this testCase | | | ### `testSuite` From 01a40e555eccb4d57494b7d85b61c05d8760cda0 Mon Sep 17 00:00:00 2001 From: olensmar Date: Fri, 14 Apr 2023 13:46:44 +0200 Subject: [PATCH 30/37] updated schema version to 0.3.0-draft --- examples/testcaserun_finished.json | 2 +- examples/testcaserun_queued.json | 2 +- examples/testcaserun_started.json | 2 +- examples/testoutput_published.json | 2 +- examples/testsuiterun_finished.json | 2 +- examples/testsuiterun_queued.json | 2 +- examples/testsuiterun_started.json | 2 +- schemas/testcaserunfinished.json | 6 +++--- schemas/testcaserunqueued.json | 6 +++--- schemas/testcaserunstarted.json | 6 +++--- schemas/testoutputpublished.json | 6 +++--- schemas/testsuiterunfinished.json | 6 +++--- schemas/testsuiterunqueued.json | 6 +++--- schemas/testsuiterunstarted.json | 6 +++--- testing-events.md | 14 +++++++------- 15 files changed, 35 insertions(+), 35 deletions(-) diff --git a/examples/testcaserun_finished.json b/examples/testcaserun_finished.json index 17158617..2ffb2916 100644 --- a/examples/testcaserun_finished.json +++ b/examples/testcaserun_finished.json @@ -3,7 +3,7 @@ "version": "0.2.0", "id": "271069a8-fc18-44f1-b38f-9d70a1695819", "source": "/event/source/123", - "type": "dev.cdevents.testcaserun.finished.0.1.0-draft", + "type": "dev.cdevents.testcaserun.finished.0.3.0-draft", "timestamp": "2023-03-20T14:27:05.315384Z" }, "subject": { diff --git a/examples/testcaserun_queued.json b/examples/testcaserun_queued.json index 71d5c12d..495f7fab 100644 --- a/examples/testcaserun_queued.json +++ b/examples/testcaserun_queued.json @@ -3,7 +3,7 @@ "version": "0.2.0", "id": "271069a8-fc18-44f1-b38f-9d70a1695819", "source": "/event/source/123", - "type": "dev.cdevents.testcaserun.queued.0.1.0-draft", + "type": "dev.cdevents.testcaserun.queued.0.3.0-draft", "timestamp": "2023-03-20T14:27:05.315384Z" }, "subject": { diff --git a/examples/testcaserun_started.json b/examples/testcaserun_started.json index 25b736a2..261dd8e4 100644 --- a/examples/testcaserun_started.json +++ b/examples/testcaserun_started.json @@ -3,7 +3,7 @@ "version": "0.2.0", "id": "271069a8-fc18-44f1-b38f-9d70a1695819", "source": "/event/source/123", - "type": "dev.cdevents.testcaserun.started.0.1.0-draft", + "type": "dev.cdevents.testcaserun.started.0.3.0-draft", "timestamp": "2023-03-20T14:27:05.315384Z" }, "subject": { diff --git a/examples/testoutput_published.json b/examples/testoutput_published.json index 9e54572e..2d52879e 100644 --- a/examples/testoutput_published.json +++ b/examples/testoutput_published.json @@ -3,7 +3,7 @@ "version": "0.2.0", "id": "271069a8-fc18-44f1-b38f-9d70a1695819", "source": "/event/source/123", - "type": "dev.cdevents.testoutput.published.0.1.0-draft", + "type": "dev.cdevents.testoutput.published.0.3.0-draft", "timestamp": "2023-03-20T14:27:05.315384Z" }, "subject": { diff --git a/examples/testsuiterun_finished.json b/examples/testsuiterun_finished.json index 0d41d42e..2864791c 100644 --- a/examples/testsuiterun_finished.json +++ b/examples/testsuiterun_finished.json @@ -3,7 +3,7 @@ "version": "0.2.0", "id": "271069a8-fc18-44f1-b38f-9d70a1695819", "source": "/event/source/123", - "type": "dev.cdevents.testsuiterun.finished.0.1.0-draft", + "type": "dev.cdevents.testsuiterun.finished.0.3.0-draft", "timestamp": "2023-03-20T14:27:05.315384Z" }, "subject": { diff --git a/examples/testsuiterun_queued.json b/examples/testsuiterun_queued.json index 1b6eb1e9..71e77eef 100644 --- a/examples/testsuiterun_queued.json +++ b/examples/testsuiterun_queued.json @@ -3,7 +3,7 @@ "version": "0.2.0", "id": "271069a8-fc18-44f1-b38f-9d70a1695819", "source": "/event/source/123", - "type": "dev.cdevents.testsuiterun.queued.0.1.0-draft", + "type": "dev.cdevents.testsuiterun.queued.0.3.0-draft", "timestamp": "2023-03-20T14:27:05.315384Z" }, "subject": { diff --git a/examples/testsuiterun_started.json b/examples/testsuiterun_started.json index 357278a6..6f459fbf 100644 --- a/examples/testsuiterun_started.json +++ b/examples/testsuiterun_started.json @@ -3,7 +3,7 @@ "version": "0.2.0", "id": "271069a8-fc18-44f1-b38f-9d70a1695819", "source": "/event/source/123", - "type": "dev.cdevents.testsuiterun.started.0.1.0-draft", + "type": "dev.cdevents.testsuiterun.started.0.3.0-draft", "timestamp": "2023-03-20T14:27:05.315384Z" }, "subject": { diff --git a/schemas/testcaserunfinished.json b/schemas/testcaserunfinished.json index b35d07dc..ea4ade1b 100644 --- a/schemas/testcaserunfinished.json +++ b/schemas/testcaserunfinished.json @@ -1,6 +1,6 @@ { "$schema": "https://json-schema.org/draft/2020-12/schema", - "$id": "https://cdevents.dev/0.1.0-draft/schema/test-case-run-finished-event", + "$id": "https://cdevents.dev/0.3.0-draft/schema/test-case-run-finished-event", "properties": { "context": { "properties": { @@ -19,9 +19,9 @@ "type": { "type": "string", "enum": [ - "dev.cdevents.testcaserun.finished.0.1.0-draft" + "dev.cdevents.testcaserun.finished.0.3.0-draft" ], - "default": "dev.cdevents.testcaserun.finished.0.1.0-draft" + "default": "dev.cdevents.testcaserun.finished.0.3.0-draft" }, "timestamp": { "type": "string", diff --git a/schemas/testcaserunqueued.json b/schemas/testcaserunqueued.json index ec6f0746..f303c90b 100644 --- a/schemas/testcaserunqueued.json +++ b/schemas/testcaserunqueued.json @@ -1,6 +1,6 @@ { "$schema": "https://json-schema.org/draft/2020-12/schema", - "$id": "https://cdevents.dev/0.1.0-draft/schema/test-case-run-queued-event", + "$id": "https://cdevents.dev/0.3.0-draft/schema/test-case-run-queued-event", "properties": { "context": { "properties": { @@ -19,9 +19,9 @@ "type": { "type": "string", "enum": [ - "dev.cdevents.testcaserun.queued.0.1.0-draft" + "dev.cdevents.testcaserun.queued.0.3.0-draft" ], - "default": "dev.cdevents.testcaserun.queued.0.1.0-draft" + "default": "dev.cdevents.testcaserun.queued.0.3.0-draft" }, "timestamp": { "type": "string", diff --git a/schemas/testcaserunstarted.json b/schemas/testcaserunstarted.json index 538bcd09..f63c2a53 100644 --- a/schemas/testcaserunstarted.json +++ b/schemas/testcaserunstarted.json @@ -1,6 +1,6 @@ { "$schema": "https://json-schema.org/draft/2020-12/schema", - "$id": "https://cdevents.dev/0.1.0-draft/schema/test-case-run-started-event", + "$id": "https://cdevents.dev/0.3.0-draft/schema/test-case-run-started-event", "properties": { "context": { "properties": { @@ -19,9 +19,9 @@ "type": { "type": "string", "enum": [ - "dev.cdevents.testcaserun.started.0.1.0-draft" + "dev.cdevents.testcaserun.started.0.3.0-draft" ], - "default": "dev.cdevents.testcaserun.started.0.1.0-draft" + "default": "dev.cdevents.testcaserun.started.0.3.0-draft" }, "timestamp": { "type": "string", diff --git a/schemas/testoutputpublished.json b/schemas/testoutputpublished.json index f2959220..025e43e3 100644 --- a/schemas/testoutputpublished.json +++ b/schemas/testoutputpublished.json @@ -1,6 +1,6 @@ { "$schema": "https://json-schema.org/draft/2020-12/schema", - "$id": "https://cdevents.dev/0.1.0-draft/schema/test-output-published-event", + "$id": "https://cdevents.dev/0.3.0-draft/schema/test-output-published-event", "properties": { "context": { "properties": { @@ -19,9 +19,9 @@ "type": { "type": "string", "enum": [ - "dev.cdevents.testoutput.published.0.1.0-draft" + "dev.cdevents.testoutput.published.0.3.0-draft" ], - "default": "dev.cdevents.testoutput.published.0.1.0-draft" + "default": "dev.cdevents.testoutput.published.0.3.0-draft" }, "timestamp": { "type": "string", diff --git a/schemas/testsuiterunfinished.json b/schemas/testsuiterunfinished.json index 9afe47c3..bd5dde98 100644 --- a/schemas/testsuiterunfinished.json +++ b/schemas/testsuiterunfinished.json @@ -1,6 +1,6 @@ { "$schema": "https://json-schema.org/draft/2020-12/schema", - "$id": "https://cdevents.dev/0.1.0-draft/schema/test-suite-finished-event", + "$id": "https://cdevents.dev/0.3.0-draft/schema/test-suite-finished-event", "properties": { "context": { "properties": { @@ -19,9 +19,9 @@ "type": { "type": "string", "enum": [ - "dev.cdevents.testsuiterun.finished.0.1.0-draft" + "dev.cdevents.testsuiterun.finished.0.3.0-draft" ], - "default": "dev.cdevents.testsuiterun.finished.0.1.0-draft" + "default": "dev.cdevents.testsuiterun.finished.0.3.0-draft" }, "timestamp": { "type": "string", diff --git a/schemas/testsuiterunqueued.json b/schemas/testsuiterunqueued.json index be4323ce..8d055d0c 100644 --- a/schemas/testsuiterunqueued.json +++ b/schemas/testsuiterunqueued.json @@ -1,6 +1,6 @@ { "$schema": "https://json-schema.org/draft/2020-12/schema", - "$id": "https://cdevents.dev/0.1.0-draft/schema/test-suite-run-queued-event", + "$id": "https://cdevents.dev/0.3.0-draft/schema/test-suite-run-queued-event", "properties": { "context": { "properties": { @@ -19,9 +19,9 @@ "type": { "type": "string", "enum": [ - "dev.cdevents.testsuiterun.queued.0.1.0-draft" + "dev.cdevents.testsuiterun.queued.0.3.0-draft" ], - "default": "dev.cdevents.testsuiterun.queued.0.1.0-draft" + "default": "dev.cdevents.testsuiterun.queued.0.3.0-draft" }, "timestamp": { "type": "string", diff --git a/schemas/testsuiterunstarted.json b/schemas/testsuiterunstarted.json index a8cc3835..38ca278c 100644 --- a/schemas/testsuiterunstarted.json +++ b/schemas/testsuiterunstarted.json @@ -1,6 +1,6 @@ { "$schema": "https://json-schema.org/draft/2020-12/schema", - "$id": "https://cdevents.dev/0.1.0-draft/schema/test-suite-run-started-event", + "$id": "https://cdevents.dev/0.3.0-draft/schema/test-suite-run-started-event", "properties": { "context": { "properties": { @@ -19,9 +19,9 @@ "type": { "type": "string", "enum": [ - "dev.cdevents.testsuiterun.started.0.1.0-draft" + "dev.cdevents.testsuiterun.started.0.3.0-draft" ], - "default": "dev.cdevents.testsuiterun.started.0.1.0-draft" + "default": "dev.cdevents.testsuiterun.started.0.3.0-draft" }, "timestamp": { "type": "string", diff --git a/testing-events.md b/testing-events.md index b2b1193e..bccf0916 100644 --- a/testing-events.md +++ b/testing-events.md @@ -69,7 +69,7 @@ One or more `testOutput` artifacts are usually produced as the result of a test This event represents when a testCaseRun has been queued for execution - and is waiting for applicable preconditions (resource availability, other tasks, etc.) to be fulfilled before actually executing. -- Event Type: __`dev.cdevents.testcaserun.queued.0.1.0-draft`__ +- Event Type: __`dev.cdevents.testcaserun.queued.0.3.0-draft`__ - Predicate: queued - Subject: [`testCaseRun`](#testcaserun) @@ -86,7 +86,7 @@ This event represents when a testCaseRun has been queued for execution - and is This event represents a started testCase execution. -- Event Type: __`dev.cdevents.testcaserun.started.0.1.0-draft`__ +- Event Type: __`dev.cdevents.testcaserun.started.0.3.0-draft`__ - Predicate: started - Subject: [`testCaseRun`](#testcaserun) @@ -103,7 +103,7 @@ This event represents a started testCase execution. This event represents a finished testCase execution. The event will contain the finished status and additional metadata as applicable. -- Event Type: __`dev.cdevents.testcaserun.finished.0.1.0-draft`__ +- Event Type: __`dev.cdevents.testcaserun.finished.0.3.0-draft`__ - Predicate: finished - Subject: [`testCaseRun`](#testcaserun) @@ -123,7 +123,7 @@ This event represents a finished testCase execution. The event will contain the This event represents when a testSuiteRun has been queued for execution - and is waiting for applicable preconditions (resource availability, other tasks, etc.) to be met before actually executing. -- Event Type: __`dev.cdevents.testsuiterun.queued.0.1.0-draft`__ +- Event Type: __`dev.cdevents.testsuiterun.queued.0.3.0-draft`__ - Predicate: queued - Subject: [`testSuiteRun`](#testsuiterun) @@ -139,7 +139,7 @@ This event represents when a testSuiteRun has been queued for execution - and is This event represents a started testSuite execution. -- Event Type: __`dev.cdevents.testsuiterun.started.0.1.0-draft`__ +- Event Type: __`dev.cdevents.testsuiterun.started.0.3.0-draft`__ - Predicate: started - Subject: [`testSuiteRun`](#testsuiterun) @@ -155,7 +155,7 @@ This event represents a started testSuite execution. This event represents a finished testSuite execution. The event will contain the execution status and additional metadata as applicable. -- Event Type: __`dev.cdevents.testsuiterun.finished.0.1.0-draft`__ +- Event Type: __`dev.cdevents.testsuiterun.finished.0.3.0-draft`__ - Predicate: finished - Subject: [`testSuiteRun`](#testsuiterun) @@ -173,7 +173,7 @@ This event represents a finished testSuite execution. The event will contain the The event represents a test execution output artifact that has been published. -- Event Type: __`dev.cdevents.testoutput.published.0.1.0-draft`__ +- Event Type: __`dev.cdevents.testoutput.published.0.3.0-draft`__ - Predicate: published - Subject: [`testOutput`](#testoutput) From 5316b7673589300fddc27905d743da7bd4a160f8 Mon Sep 17 00:00:00 2001 From: olensmar Date: Mon, 17 Apr 2023 14:16:09 +0200 Subject: [PATCH 31/37] renamed status to outcome --- examples/testcaserun_finished.json | 2 +- examples/testsuiterun_finished.json | 2 +- schemas/testcaserunfinished.json | 2 +- schemas/testsuiterunfinished.json | 2 +- testing-events.md | 42 ++++++++++++++--------------- 5 files changed, 25 insertions(+), 25 deletions(-) diff --git a/examples/testcaserun_finished.json b/examples/testcaserun_finished.json index 2ffb2916..4382b03e 100644 --- a/examples/testcaserun_finished.json +++ b/examples/testcaserun_finished.json @@ -11,7 +11,7 @@ "source": "/event/source/123", "type": "testCaseRun", "content": { - "status" : "pass", + "outcome" : "pass", "environment": { "id": "dev", "source": "testkube-dev-123" diff --git a/examples/testsuiterun_finished.json b/examples/testsuiterun_finished.json index 2864791c..2e30a842 100644 --- a/examples/testsuiterun_finished.json +++ b/examples/testsuiterun_finished.json @@ -11,7 +11,7 @@ "source": "/event/source/123", "type": "testSuiteRun", "content": { - "status" : "fail", + "outcome" : "fail", "severity" : "critical", "reason" : "Host 123.34.23.32 not found", "environment": { diff --git a/schemas/testcaserunfinished.json b/schemas/testcaserunfinished.json index ea4ade1b..fb9164c7 100644 --- a/schemas/testcaserunfinished.json +++ b/schemas/testcaserunfinished.json @@ -57,7 +57,7 @@ }, "content": { "properties": { - "status": { + "outcome": { "type": "string", "enum": [ "pass", diff --git a/schemas/testsuiterunfinished.json b/schemas/testsuiterunfinished.json index bd5dde98..df7339b3 100644 --- a/schemas/testsuiterunfinished.json +++ b/schemas/testsuiterunfinished.json @@ -98,7 +98,7 @@ } } }, - "status": { + "outcome": { "type": "string", "enum": [ "pass", diff --git a/testing-events.md b/testing-events.md index bccf0916..17e7bfb6 100644 --- a/testing-events.md +++ b/testing-events.md @@ -101,22 +101,22 @@ This event represents a started testCase execution. ### `testCaseRun finished` -This event represents a finished testCase execution. The event will contain the finished status and additional metadata as applicable. +This event represents a finished testCase execution. The event will contain the outcome and additional metadata as applicable. - Event Type: __`dev.cdevents.testcaserun.finished.0.3.0-draft`__ - Predicate: finished - Subject: [`testCaseRun`](#testcaserun) -| Field | Type | Description | Examples | Required | -|--------------|-----------------------------------------------------------------|---------------------------------------------------------------------------------|-------------------------------------------------------------------------|----------| -| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | -| environment | `Object` [`environment`](continuous-deployment.md/#environment) | The environment in which this testCaseRun was running | `{"id": "1234"}`, `{"id": "dev", "source": "testkube-dev-123"}` | ✅ | -| testCase | `Object` [`testCase`](#testcase) | Definition of the testCase being executed | `{"id": "92834723894", "name": "Login Test", "type": "integration"}` | | -| testSuiteRun | `Object` [`testSuiteRun`](#testsuiterun) | A testSuiteRun to associate this testCaseRun with a containing testSuiteRun | `{"id":"Auth-TestSuite-execution-12334", "source": "staging/testkube"}` | | -| status | `String` | The status of the testSuite execution, one of `pass`, `fail`, `cancel`, `error` | `pass` | ✅ | -| severity | `String` | Severity if the test failed, one of `low`, `medium`, `high`, `critical` | `critical` | -| reason | `String` | A reason related to the `status` of the execution | `Cancelled by user`, `Failed assertion`, `Timed out` | | +| Field | Type | Description | Examples | Required | +|--------------|-----------------------------------------------------------------|----------------------------------------------------------------------------------|-------------------------------------------------------------------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `unitest-abc`, `e2e-test1`, `scan-image1` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| environment | `Object` [`environment`](continuous-deployment.md/#environment) | The environment in which this testCaseRun was running | `{"id": "1234"}`, `{"id": "dev", "source": "testkube-dev-123"}` | ✅ | +| testCase | `Object` [`testCase`](#testcase) | Definition of the testCase being executed | `{"id": "92834723894", "name": "Login Test", "type": "integration"}` | | +| testSuiteRun | `Object` [`testSuiteRun`](#testsuiterun) | A testSuiteRun to associate this testCaseRun with a containing testSuiteRun | `{"id":"Auth-TestSuite-execution-12334", "source": "staging/testkube"}` | | +| outcome | `String` | The outcome of the testSuite execution, one of `pass`, `fail`, `cancel`, `error` | `pass` | ✅ | +| severity | `String` | Severity if the test failed, one of `low`, `medium`, `high`, `critical` | `critical` | +| reason | `String` | A reason related to the outcome of the execution | `Cancelled by user`, `Failed assertion`, `Timed out` | | ### `testSuiteRun queued` @@ -153,21 +153,21 @@ This event represents a started testSuite execution. ### `testSuiteRun finished` -This event represents a finished testSuite execution. The event will contain the execution status and additional metadata as applicable. +This event represents a finished testSuite execution. The event will contain the outcome and additional metadata as applicable. - Event Type: __`dev.cdevents.testsuiterun.finished.0.3.0-draft`__ - Predicate: finished - Subject: [`testSuiteRun`](#testsuiterun) -| Field | Type | Description | Examples | Required | -|-------------|-----------------------------------------------------------------|---------------------------------------------------------------------------------|-----------------------------------------------------------------|----------| -| id | `String` | Uniquely identifies the subject within the source. | `unit`, `e2e`, `security` | ✅ | -| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | -| environment | `Object` [`environment`](continuous-deployment.md/#environment) | The environment in which this testSuiteRun was running | `{"id": "1234"}`, `{"id": "dev", "source": "testkube-dev-123"}` | ✅ | -| status | `String` | The status of the testSuite execution, one of `pass`, `fail`, `cancel`, `error` | `fail` | ✅ | -| severity | `String` | Severity if the test failed, one of `low`, `medium`, `high`, `critical` | `critical`, `low`, `medium`, `high` | -| reason | `String` | A reason related to the status of the execution | `Cancelled by user`, `Failed testCase` | | -| testSuite | `Object` [`testSuite`](#testsuite) | Definition of the testSuite being executed | `{"id": "92834723894", "name": "Auth TestSuite"}` | | +| Field | Type | Description | Examples | Required | +|-------------|-----------------------------------------------------------------|----------------------------------------------------------------------------------|-----------------------------------------------------------------|----------| +| id | `String` | Uniquely identifies the subject within the source. | `unit`, `e2e`, `security` | ✅ | +| source | `URI-Reference` | [source](spec.md#source--context-) from the context | | | +| environment | `Object` [`environment`](continuous-deployment.md/#environment) | The environment in which this testSuiteRun was running | `{"id": "1234"}`, `{"id": "dev", "source": "testkube-dev-123"}` | ✅ | +| outcome | `String` | The outcome of the testSuite execution, one of `pass`, `fail`, `cancel`, `error` | `fail` | ✅ | +| severity | `String` | Severity if the test failed, one of `low`, `medium`, `high`, `critical` | `critical`, `low`, `medium`, `high` | +| reason | `String` | A reason related to the outcome of the execution | `Cancelled by user`, `Failed testCase` | | +| testSuite | `Object` [`testSuite`](#testsuite) | Definition of the testSuite being executed | `{"id": "92834723894", "name": "Auth TestSuite"}` | | ### `testOutput published` From d38ac16b602cf80b08c35f7491a3bdb69cc1befb Mon Sep 17 00:00:00 2001 From: olensmar Date: Mon, 17 Apr 2023 14:48:59 +0200 Subject: [PATCH 32/37] removed type from testCaseRun --- testing-events.md | 1 - 1 file changed, 1 deletion(-) diff --git a/testing-events.md b/testing-events.md index 17e7bfb6..1cab5e11 100644 --- a/testing-events.md +++ b/testing-events.md @@ -33,7 +33,6 @@ A `testCaseRun` is the smallest unit of testing that the user wants to track, `t |--------------|-----------------------------------------------------------------|---------------------------------------------------------------------------------------|-------------------------------------------------------------------------| | id | `String` | Uniquely identifies the subject within the source. | `Login-Test-execution-1213423`, `e2e-test1`, `2334234324` | | source | `URI-Reference` | [source](spec.md#source--context-) from the context | `staging/testkube`, `testkube-dev-123` | -| type | `String` | An optional type of test | `functional`, `unit`, `performance`, `security` | | environment | `Object` [`environment`](continuous-deployment.md/#environment) | The environment in which this testCaseRun is executing | `{"id": "1234"}`, `{"id": "dev", "source": "testkube-dev-123"}` | | testCase | `Object` [`testCase`](#testcase) | An optional definition of the testCase being executed | `{"id": "92834723894", "name": "Login Test", "type": "integration"}` | | | testSuiteRun | `Object` [`testSuiteRun`](#testsuiterun) | An optional testSuiteRun to associate this testCaseRun with a containing testSuiteRun | `{"id":"Auth-TestSuite-execution-12334", "source": "staging/testkube"}` | | From aa13c88f5ec9d022760e023db1d9d900b5ef6fec Mon Sep 17 00:00:00 2001 From: olensmar Date: Tue, 18 Apr 2023 13:41:31 +0200 Subject: [PATCH 33/37] fixed versions and outputType --- examples/testcaserun_finished.json | 4 ++-- examples/testcaserun_queued.json | 4 ++-- examples/testcaserun_started.json | 4 ++-- examples/testoutput_published.json | 4 ++-- examples/testsuiterun_finished.json | 4 ++-- examples/testsuiterun_queued.json | 4 ++-- examples/testsuiterun_started.json | 4 ++-- schemas/testcaserunfinished.json | 4 ++-- schemas/testcaserunqueued.json | 4 ++-- schemas/testcaserunstarted.json | 4 ++-- schemas/testoutputpublished.json | 4 ++-- schemas/testsuiterunfinished.json | 4 ++-- schemas/testsuiterunqueued.json | 4 ++-- schemas/testsuiterunstarted.json | 4 ++-- testing-events.md | 18 +++++++++--------- 15 files changed, 37 insertions(+), 37 deletions(-) diff --git a/examples/testcaserun_finished.json b/examples/testcaserun_finished.json index 4382b03e..2a349a23 100644 --- a/examples/testcaserun_finished.json +++ b/examples/testcaserun_finished.json @@ -1,9 +1,9 @@ { "context": { - "version": "0.2.0", + "version": "0.3.0-draft", "id": "271069a8-fc18-44f1-b38f-9d70a1695819", "source": "/event/source/123", - "type": "dev.cdevents.testcaserun.finished.0.3.0-draft", + "type": "dev.cdevents.testcaserun.finished.0.1.0-draft", "timestamp": "2023-03-20T14:27:05.315384Z" }, "subject": { diff --git a/examples/testcaserun_queued.json b/examples/testcaserun_queued.json index 495f7fab..6054b78c 100644 --- a/examples/testcaserun_queued.json +++ b/examples/testcaserun_queued.json @@ -1,9 +1,9 @@ { "context": { - "version": "0.2.0", + "version": "0.3.0-draft", "id": "271069a8-fc18-44f1-b38f-9d70a1695819", "source": "/event/source/123", - "type": "dev.cdevents.testcaserun.queued.0.3.0-draft", + "type": "dev.cdevents.testcaserun.queued.0.1.0-draft", "timestamp": "2023-03-20T14:27:05.315384Z" }, "subject": { diff --git a/examples/testcaserun_started.json b/examples/testcaserun_started.json index 261dd8e4..d48f5d69 100644 --- a/examples/testcaserun_started.json +++ b/examples/testcaserun_started.json @@ -1,9 +1,9 @@ { "context": { - "version": "0.2.0", + "version": "0.3.0-draft", "id": "271069a8-fc18-44f1-b38f-9d70a1695819", "source": "/event/source/123", - "type": "dev.cdevents.testcaserun.started.0.3.0-draft", + "type": "dev.cdevents.testcaserun.started.0.1.0-draft", "timestamp": "2023-03-20T14:27:05.315384Z" }, "subject": { diff --git a/examples/testoutput_published.json b/examples/testoutput_published.json index 2d52879e..7aa2fbe9 100644 --- a/examples/testoutput_published.json +++ b/examples/testoutput_published.json @@ -1,9 +1,9 @@ { "context": { - "version": "0.2.0", + "version": "0.3.0-draft", "id": "271069a8-fc18-44f1-b38f-9d70a1695819", "source": "/event/source/123", - "type": "dev.cdevents.testoutput.published.0.3.0-draft", + "type": "dev.cdevents.testoutput.published.0.1.0-draft", "timestamp": "2023-03-20T14:27:05.315384Z" }, "subject": { diff --git a/examples/testsuiterun_finished.json b/examples/testsuiterun_finished.json index 2e30a842..7a4d1021 100644 --- a/examples/testsuiterun_finished.json +++ b/examples/testsuiterun_finished.json @@ -1,9 +1,9 @@ { "context": { - "version": "0.2.0", + "version": "0.3.0-draft", "id": "271069a8-fc18-44f1-b38f-9d70a1695819", "source": "/event/source/123", - "type": "dev.cdevents.testsuiterun.finished.0.3.0-draft", + "type": "dev.cdevents.testsuiterun.finished.0.1.0-draft", "timestamp": "2023-03-20T14:27:05.315384Z" }, "subject": { diff --git a/examples/testsuiterun_queued.json b/examples/testsuiterun_queued.json index 71e77eef..fe3d1ee7 100644 --- a/examples/testsuiterun_queued.json +++ b/examples/testsuiterun_queued.json @@ -1,9 +1,9 @@ { "context": { - "version": "0.2.0", + "version": "0.3.0-draft", "id": "271069a8-fc18-44f1-b38f-9d70a1695819", "source": "/event/source/123", - "type": "dev.cdevents.testsuiterun.queued.0.3.0-draft", + "type": "dev.cdevents.testsuiterun.queued.0.1.0-draft", "timestamp": "2023-03-20T14:27:05.315384Z" }, "subject": { diff --git a/examples/testsuiterun_started.json b/examples/testsuiterun_started.json index 6f459fbf..4c87895d 100644 --- a/examples/testsuiterun_started.json +++ b/examples/testsuiterun_started.json @@ -1,9 +1,9 @@ { "context": { - "version": "0.2.0", + "version": "0.3.0-draft", "id": "271069a8-fc18-44f1-b38f-9d70a1695819", "source": "/event/source/123", - "type": "dev.cdevents.testsuiterun.started.0.3.0-draft", + "type": "dev.cdevents.testsuiterun.started.0.1.0-draft", "timestamp": "2023-03-20T14:27:05.315384Z" }, "subject": { diff --git a/schemas/testcaserunfinished.json b/schemas/testcaserunfinished.json index fb9164c7..18748001 100644 --- a/schemas/testcaserunfinished.json +++ b/schemas/testcaserunfinished.json @@ -19,9 +19,9 @@ "type": { "type": "string", "enum": [ - "dev.cdevents.testcaserun.finished.0.3.0-draft" + "dev.cdevents.testcaserun.finished.0.1.0-draft" ], - "default": "dev.cdevents.testcaserun.finished.0.3.0-draft" + "default": "dev.cdevents.testcaserun.finished.0.1.0-draft" }, "timestamp": { "type": "string", diff --git a/schemas/testcaserunqueued.json b/schemas/testcaserunqueued.json index f303c90b..e8226ec9 100644 --- a/schemas/testcaserunqueued.json +++ b/schemas/testcaserunqueued.json @@ -19,9 +19,9 @@ "type": { "type": "string", "enum": [ - "dev.cdevents.testcaserun.queued.0.3.0-draft" + "dev.cdevents.testcaserun.queued.0.1.0-draft" ], - "default": "dev.cdevents.testcaserun.queued.0.3.0-draft" + "default": "dev.cdevents.testcaserun.queued.0.1.0-draft" }, "timestamp": { "type": "string", diff --git a/schemas/testcaserunstarted.json b/schemas/testcaserunstarted.json index f63c2a53..1a12bd1f 100644 --- a/schemas/testcaserunstarted.json +++ b/schemas/testcaserunstarted.json @@ -19,9 +19,9 @@ "type": { "type": "string", "enum": [ - "dev.cdevents.testcaserun.started.0.3.0-draft" + "dev.cdevents.testcaserun.started.0.1.0-draft" ], - "default": "dev.cdevents.testcaserun.started.0.3.0-draft" + "default": "dev.cdevents.testcaserun.started.0.1.0-draft" }, "timestamp": { "type": "string", diff --git a/schemas/testoutputpublished.json b/schemas/testoutputpublished.json index 025e43e3..61997f42 100644 --- a/schemas/testoutputpublished.json +++ b/schemas/testoutputpublished.json @@ -19,9 +19,9 @@ "type": { "type": "string", "enum": [ - "dev.cdevents.testoutput.published.0.3.0-draft" + "dev.cdevents.testoutput.published.0.1.0-draft" ], - "default": "dev.cdevents.testoutput.published.0.3.0-draft" + "default": "dev.cdevents.testoutput.published.0.1.0-draft" }, "timestamp": { "type": "string", diff --git a/schemas/testsuiterunfinished.json b/schemas/testsuiterunfinished.json index df7339b3..ab5dbbce 100644 --- a/schemas/testsuiterunfinished.json +++ b/schemas/testsuiterunfinished.json @@ -19,9 +19,9 @@ "type": { "type": "string", "enum": [ - "dev.cdevents.testsuiterun.finished.0.3.0-draft" + "dev.cdevents.testsuiterun.finished.0.1.0-draft" ], - "default": "dev.cdevents.testsuiterun.finished.0.3.0-draft" + "default": "dev.cdevents.testsuiterun.finished.0.1.0-draft" }, "timestamp": { "type": "string", diff --git a/schemas/testsuiterunqueued.json b/schemas/testsuiterunqueued.json index 8d055d0c..1b433945 100644 --- a/schemas/testsuiterunqueued.json +++ b/schemas/testsuiterunqueued.json @@ -19,9 +19,9 @@ "type": { "type": "string", "enum": [ - "dev.cdevents.testsuiterun.queued.0.3.0-draft" + "dev.cdevents.testsuiterun.queued.0.1.0-draft" ], - "default": "dev.cdevents.testsuiterun.queued.0.3.0-draft" + "default": "dev.cdevents.testsuiterun.queued.0.1.0-draft" }, "timestamp": { "type": "string", diff --git a/schemas/testsuiterunstarted.json b/schemas/testsuiterunstarted.json index 38ca278c..4b6814d9 100644 --- a/schemas/testsuiterunstarted.json +++ b/schemas/testsuiterunstarted.json @@ -19,9 +19,9 @@ "type": { "type": "string", "enum": [ - "dev.cdevents.testsuiterun.started.0.3.0-draft" + "dev.cdevents.testsuiterun.started.0.1.0-draft" ], - "default": "dev.cdevents.testsuiterun.started.0.3.0-draft" + "default": "dev.cdevents.testsuiterun.started.0.1.0-draft" }, "timestamp": { "type": "string", diff --git a/testing-events.md b/testing-events.md index 1cab5e11..b69f3f88 100644 --- a/testing-events.md +++ b/testing-events.md @@ -56,7 +56,7 @@ One or more `testOutput` artifacts are usually produced as the result of a test |-------------|----------------------------------------|---------------------------------------------------------------------------|-----------------------------------------------------------------------| | id | `String` | Uniquely identifies the subject within the source. | `report-23123123` | | source | `URI-Reference` | [source](spec.md#source--context-) from the context | `staging/testkube`, `testkube-dev-123` | -| type | `String` | The type of output, one of `report`, `video`, `image`, `log`, `other` | `video` | +| outputType | `String` | The type of output, one of `report`, `video`, `image`, `log`, `other` | `video` | | format | `String` | The Content-Type of the output artifact | `application/pdf`, `image/png`, `application/json` | | uri | `URI-Reference` | A reference to retrieve the specified output artifact | `https://testkube.mycluster.internal/artifacts/23123123` | | testCaseRun | `Object` [`testCaseRun`](#testcaserun) | An optional testCaseRun to link this artifact to a specific `testCaseRun` | `{"id":"Login-Test-execution-1213423", "source": "staging/testkube"}` | @@ -68,7 +68,7 @@ One or more `testOutput` artifacts are usually produced as the result of a test This event represents when a testCaseRun has been queued for execution - and is waiting for applicable preconditions (resource availability, other tasks, etc.) to be fulfilled before actually executing. -- Event Type: __`dev.cdevents.testcaserun.queued.0.3.0-draft`__ +- Event Type: __`dev.cdevents.testcaserun.queued.0.1.0-draft`__ - Predicate: queued - Subject: [`testCaseRun`](#testcaserun) @@ -85,7 +85,7 @@ This event represents when a testCaseRun has been queued for execution - and is This event represents a started testCase execution. -- Event Type: __`dev.cdevents.testcaserun.started.0.3.0-draft`__ +- Event Type: __`dev.cdevents.testcaserun.started.0.1.0-draft`__ - Predicate: started - Subject: [`testCaseRun`](#testcaserun) @@ -102,7 +102,7 @@ This event represents a started testCase execution. This event represents a finished testCase execution. The event will contain the outcome and additional metadata as applicable. -- Event Type: __`dev.cdevents.testcaserun.finished.0.3.0-draft`__ +- Event Type: __`dev.cdevents.testcaserun.finished.0.1.0-draft`__ - Predicate: finished - Subject: [`testCaseRun`](#testcaserun) @@ -122,7 +122,7 @@ This event represents a finished testCase execution. The event will contain the This event represents when a testSuiteRun has been queued for execution - and is waiting for applicable preconditions (resource availability, other tasks, etc.) to be met before actually executing. -- Event Type: __`dev.cdevents.testsuiterun.queued.0.3.0-draft`__ +- Event Type: __`dev.cdevents.testsuiterun.queued.0.1.0-draft`__ - Predicate: queued - Subject: [`testSuiteRun`](#testsuiterun) @@ -138,7 +138,7 @@ This event represents when a testSuiteRun has been queued for execution - and is This event represents a started testSuite execution. -- Event Type: __`dev.cdevents.testsuiterun.started.0.3.0-draft`__ +- Event Type: __`dev.cdevents.testsuiterun.started.0.1.0-draft`__ - Predicate: started - Subject: [`testSuiteRun`](#testsuiterun) @@ -154,7 +154,7 @@ This event represents a started testSuite execution. This event represents a finished testSuite execution. The event will contain the outcome and additional metadata as applicable. -- Event Type: __`dev.cdevents.testsuiterun.finished.0.3.0-draft`__ +- Event Type: __`dev.cdevents.testsuiterun.finished.0.1.0-draft`__ - Predicate: finished - Subject: [`testSuiteRun`](#testsuiterun) @@ -165,14 +165,14 @@ This event represents a finished testSuite execution. The event will contain the | environment | `Object` [`environment`](continuous-deployment.md/#environment) | The environment in which this testSuiteRun was running | `{"id": "1234"}`, `{"id": "dev", "source": "testkube-dev-123"}` | ✅ | | outcome | `String` | The outcome of the testSuite execution, one of `pass`, `fail`, `cancel`, `error` | `fail` | ✅ | | severity | `String` | Severity if the test failed, one of `low`, `medium`, `high`, `critical` | `critical`, `low`, `medium`, `high` | -| reason | `String` | A reason related to the outcome of the execution | `Cancelled by user`, `Failed testCase` | | +| reason | `String` | A reason related to the outcome of the execution | `Cancelled by user`, `Failed testCase` | | | testSuite | `Object` [`testSuite`](#testsuite) | Definition of the testSuite being executed | `{"id": "92834723894", "name": "Auth TestSuite"}` | | ### `testOutput published` The event represents a test execution output artifact that has been published. -- Event Type: __`dev.cdevents.testoutput.published.0.3.0-draft`__ +- Event Type: __`dev.cdevents.testoutput.published.0.1.0-draft`__ - Predicate: published - Subject: [`testOutput`](#testoutput) From 0cedc9f65ae446b554c3391bea4b4233b42ffec8 Mon Sep 17 00:00:00 2001 From: Ole Lensmar Date: Thu, 20 Apr 2023 16:20:34 +0200 Subject: [PATCH 34/37] Update testing-events.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit fixed enum example Co-authored-by: Emil Bäckmark --- testing-events.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/testing-events.md b/testing-events.md index b69f3f88..7879dddf 100644 --- a/testing-events.md +++ b/testing-events.md @@ -203,7 +203,7 @@ A `testCase` is the actual test that is being run by a `testCaseRun`. | Field | Type | Description | Examples | Required | |---------|-----------------|-----------------------------------------------------------------------------------------------------------------------|----------------------------|----------| | id | `String` | Uniquely identifies the testCase within a test management system. | `12312334` | ✅ | -| type | `String enum` | The type of test, one of `performance`, `functional`, `unit`, `security`, `compliance`, `integration`, `e2e`, `other` | `functional`,`performance` | | +| type | `String enum` | The type of test, one of `performance`, `functional`, `unit`, `security`, `compliance`, `integration`, `e2e`, `other` | `functional` | | | name | `String` | A user-friendly name for this testCase | `Login Test` | | | version | `String` | The version of the testCase | `1.0` | | | uri | `URI-Reference` | A uri reference to this testCase | | | From 059a9decd057121ab1bea8bb061ccd7dc91e3e9f Mon Sep 17 00:00:00 2001 From: olensmar Date: Thu, 20 Apr 2023 16:22:59 +0200 Subject: [PATCH 35/37] fixed wording --- testing-events.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/testing-events.md b/testing-events.md index b69f3f88..0bb611bc 100644 --- a/testing-events.md +++ b/testing-events.md @@ -170,7 +170,7 @@ This event represents a finished testSuite execution. The event will contain the ### `testOutput published` -The event represents a test execution output artifact that has been published. +The event represents a test execution output that has been published. - Event Type: __`dev.cdevents.testoutput.published.0.1.0-draft`__ - Predicate: published From edd90294290ffe17f0f384f63923baa57015a9af Mon Sep 17 00:00:00 2001 From: olensmar Date: Fri, 21 Apr 2023 10:36:20 +0200 Subject: [PATCH 36/37] fixed required outcome property --- schemas/testcaserunfinished.json | 2 +- schemas/testsuiterunfinished.json | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/schemas/testcaserunfinished.json b/schemas/testcaserunfinished.json index 18748001..06ec9830 100644 --- a/schemas/testcaserunfinished.json +++ b/schemas/testcaserunfinished.json @@ -152,7 +152,7 @@ "additionalProperties": false, "type": "object", "required": [ - "status", + "outcome", "environment" ] } diff --git a/schemas/testsuiterunfinished.json b/schemas/testsuiterunfinished.json index ab5dbbce..c59f7198 100644 --- a/schemas/testsuiterunfinished.json +++ b/schemas/testsuiterunfinished.json @@ -123,7 +123,7 @@ "additionalProperties": false, "type": "object", "required": [ - "status", + "outcome", "environment" ] } From 3198d1739830427ae8a380b4de96d62adcb93807 Mon Sep 17 00:00:00 2001 From: olensmar Date: Fri, 21 Apr 2023 10:36:20 +0200 Subject: [PATCH 37/37] fixed required outcome property --- schemas/testcaserunfinished.json | 2 +- schemas/testsuiterunfinished.json | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/schemas/testcaserunfinished.json b/schemas/testcaserunfinished.json index 18748001..06ec9830 100644 --- a/schemas/testcaserunfinished.json +++ b/schemas/testcaserunfinished.json @@ -152,7 +152,7 @@ "additionalProperties": false, "type": "object", "required": [ - "status", + "outcome", "environment" ] } diff --git a/schemas/testsuiterunfinished.json b/schemas/testsuiterunfinished.json index ab5dbbce..c59f7198 100644 --- a/schemas/testsuiterunfinished.json +++ b/schemas/testsuiterunfinished.json @@ -123,7 +123,7 @@ "additionalProperties": false, "type": "object", "required": [ - "status", + "outcome", "environment" ] }