add definitions.md #53
Conversation
This commit adds the foundation for definitions to give a common semantics across the various concepts within the SDLC. This commit explicitly starts with 'build'. Signed-off-by: xibz <[email protected]>
✅ Deploy Preview for cdevents ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
In reference to cdevents/spec#253 |
| | Docker Registry | Stores and manages Docker container images | | ||
| | Git/GitHub/GitLab| Version Control Systems that store source code and its history | | ||
| | Amazon S3/Azure Blob Storage/GCS | Object storage services used for archiving logs, backups, or raw data | | ||
| | Kubernetes | Can store configuration via ConfigMaps/Secrets, but not typically "artifacts" | |
There was a problem hiding this comment.
I would push against the idea of Kube as a store. It is an anti-pattern to rely on Kube ConfigMaps/Secrets for durable storage. Typically "ConfigMaps/Secrets" are simply the add-on data to assist Kube to accomplish its scheduling / declarative-state tasks.
| run a system. These include: | ||
|
|
||
| - **Static assets**: images, fonts, CSS, markdown | ||
| - **Configuration files**: YAML, JSON, XML, `.env` |
|
|
||
| This document defines common concepts and ideas used across the SDLC (software | ||
| development lifecycle). By establishing a shared vocabulary, it ensures that | ||
| tools leveraging CDEvents use a consistent language and adhere to common |
There was a problem hiding this comment.
Do you want to link back to the cdevents spec?
- CD ..
- CI ...
- Testing ..
- Operations ...
There was a problem hiding this comment.
Yea, I think we need to think about whether we want to link to the spec or have the spec link to the definitions page.
|
|
||
| ## 🏗️ Build | ||
|
|
||
| A **build** is the automated process of transforming **source code** and |
There was a problem hiding this comment.
Reference cdevents spec of build ?
Related CDEvents for Builds
| - 🏷️ **Versioning**: tagging or stamping artifacts | ||
| - 🧪 **Static analysis** *(pre-build or during build)* | ||
| - 🧬 **Configuration templating**: injecting environment-specific variables | ||
|
|
There was a problem hiding this comment.
Manifest rendering
Artifact signing
| @@ -0,0 +1,208 @@ | |||
| # CDEvents Concepts and Definitions | |||
|
|
|||
| This document defines common concepts and ideas used across the SDLC (software | |||
There was a problem hiding this comment.
In the introduction, highlight that this is meant to be used as a dictionary
| | Helm template to Kubernetes | ✅ Yes | `helm template` to render manifests | Generates deployable infra config | | ||
| | Helm chart packaging | ✅ Yes | `helm package` to create `.tgz` chart | Versioned artifact used in CI/CD pipelines | | ||
| | Helm deployment | ❌ No | `helm install` or `upgrade` | Deployment, not artifact creation | | ||
|
|
There was a problem hiding this comment.
More Build Example possibilities:
- Terraform plan generation
- CloudFormation packaging
- Lambda function packaging
- Buildpack container creation
- Kustomize overlay build
- SBOM generation
- Sealed Secret encryption
There was a problem hiding this comment.
If Build is generating "excutable", then what is generating or updating "non-executable" (metadata, SBOM, signature, encrypted resources,... update dependencies's version).
There was a problem hiding this comment.
Build is generating an executable artifact with resources that may be in the actual build or during the "pre-build" process, like SBOM.
So SBOM is a resource that is used in a build, but doesn't constitute a build by itself at all. It's an artifact that can be used a resource during the build process. I can include them in the resource section, however.
| | GitHub Actions | No strict definition — "build" is defined by your workflow steps | | ||
| | GitLab CI/CD | Often treated as a named stage (e.g., `build`, `test`, `deploy`) | | ||
| | Spinnaker | Uses "bake" or "artifact" stages, and may trigger builds externally | | ||
|
|
There was a problem hiding this comment.
Other Tool Possibilities:
- AWS CodeBuild
- Azure Pipelines
- Google Cloud Build
- CircleCI / Tekton
- BuildKit/Kaniko
- Skaffold
| - **Test Cases**: Specific inputs, execution steps, and expected outputs or behaviors. | ||
| - **Test Suites**: Collections of related test cases, often grouped by functionality or type. | ||
| - **Test Data**: Data used as input for test cases (e.g., mock data, sanitized production data). | ||
| - **Test Environments**: Specific configurations of hardware, software, and network settings where tests are executed. |
There was a problem hiding this comment.
Ephemeral / Declarative / etc.. as other Environment possibilities?
| - 📝 **Reporting**: Generating logs, metrics, and reports of test results. | ||
| - ⚙️ **Setup/Teardown**: Preparing the test environment before execution and cleaning up resources afterward. | ||
| - 📈 **Coverage Analysis**: Measuring the extent to which the code or system has been exercised by tests. | ||
|
|
There was a problem hiding this comment.
Possibly add metric collection and analysis for canary/blue-green/progressive delivery?
| | Manual Exploratory Testing | ❌ No | Human tester exploring application functionality | Not an automated process, thus outside CDEvents scope | | ||
| | Code Linting/Formatting | ❌ No | `prettier --check`, `gofmt` | Focuses on code style and consistency, not functional correctness | | ||
| | Dependency Vulnerability Scan| ❌ No | `npm audit`, Snyk scan | Scans for known vulnerabilities in dependencies, not functional test of *your* code | | ||
|
|
There was a problem hiding this comment.
Other Test possibilities:
- Policy validation (OPA/Kyverno)
- chaos engineering?
- canary analysis?
- conformance testing
| | SonarQube | Primarily for static analysis and code quality, but often integrated with test results for coverage | | ||
| | Jest/JUnit/Pytest| Frameworks for writing and executing tests within codebases | | ||
| | Selenium/Cypress | Frameworks for automated browser-based functional testing | | ||
|
|
There was a problem hiding this comment.
other tool possibilities:
- Litmus/Chaos Mesh
- Flagger / Argo Rollouts
- Sonobuoy
- k6/Locust
- Testcontainers
There was a problem hiding this comment.
You forgot kubetest (iirc the contributor of testXxx in CDEvents spec) ;-)
The list is a bit of confusion, because it's a mix of test "lib/framework" and workflow tools that can launch those "lib/framework".
Maybe Test should be defined as a step that generates an analysis/report (Test Result), a "non-executable" resource. This resource can be read & used by humans or automates. It also means that those resources should be published/stored (could be ephemeral), as listed in the "Store" section.
There was a problem hiding this comment.
@davidB - The mixing is intentional because both layers are important for CDEvents adoption. Consider this scenario: A team using Jenkins might disable CDEvents generation in their JUnit tests (maybe to avoid duplication or to add custom metadata). Instead, Jenkins itself captures the JUnit results and emits CDEvents with additional context like build number, environment details, or custom tags that JUnit wouldn't know about.
This flexibility is crucial; teams need to control WHERE in their stack CDEvents are generated. Sometimes it's the test framework, sometimes it's the CI/CD tool wrapping it, sometimes both (with deduplication logic).
The table shows both perspectives because implementers need to understand how 'test' is defined at each layer, regardless of where they choose to emit events. As long as the semantics match our definition, the source doesn't matter.
| **Storing** is the automated process of persistently saving, managing, and | ||
| making accessible any **artifact, data, or metadata** generated or consumed | ||
| during the SDLC. It ensures that necessary components and information are | ||
| retrievable for future use, auditing, or deployment. |
There was a problem hiding this comment.
Alternative:
Storing is the automated process of persisting, versioning, and managing access to artifacts and metadata in durable, addressable repositories. This includes initial publication, cross-region replication, artifact promotion between environments, and lifecycle management, but excludes deployment or execution of stored artifacts.
| - **Metrics**: Performance, usage, or health data collected over time. | ||
| - **Documentation**: Generated or manually created documentation. | ||
| - **Snapshots/Backups**: Point-in-time copies of databases or systems. | ||
|
|
There was a problem hiding this comment.
Other assets?:
- SBOMs, signatures, attestations
- charts, OCI artifacts, policy bundles
- GitOps State? could be the same as VCS/Source-code?
| - 🧹 **Retention/Cleanup**: Managing the lifecycle of stored assets, including deletion policies. | ||
| - 🔄 **Replication**: Copying assets to multiple locations for redundancy or distribution. | ||
| - 🔗 **Linking**: Establishing relationships between stored assets and other SDLC events (e.g., linking an artifact to the build that produced it). | ||
|
|
There was a problem hiding this comment.
additional ideas:
- Promotion: Moving artifacts between environment-specific repositories (dev→staging→prod)
- Signing/Attestation: Adding cryptographic signatures or metadata to stored artifacts
- Scanning: Automated security/vulnerability scanning upon storage
| | Caching Build Dependencies | ❌ No | Local Maven `.m2` repository cache | Temporary storage for performance, not persistent distribution | | ||
| | Downloading a Dependency | ❌ No | `npm install` | Retrieval from storage, not the act of storing | | ||
| | Writing a Log File to Local Disk | ❌ No | `logger.info("message")` to a local file | Local/temporary persistence; "storing" implies a managed, accessible repository | | ||
|
|
There was a problem hiding this comment.
other example ideas:
- Push Helm Chart as OCI
- Cosign Image Signing
- Artifact Promotion
- GitOps Manifest Commit
- SBOM Attachment
- Multi-region Replication
- Harbor Vulnerability Scan Results
- BuildKit Cache Export
| | Helm template to Kubernetes | ✅ Yes | `helm template` to render manifests | Generates deployable infra config | | ||
| | Helm chart packaging | ✅ Yes | `helm package` to create `.tgz` chart | Versioned artifact used in CI/CD pipelines | | ||
| | Helm deployment | ❌ No | `helm install` or `upgrade` | Deployment, not artifact creation | | ||
|
|
There was a problem hiding this comment.
If Build is generating "excutable", then what is generating or updating "non-executable" (metadata, SBOM, signature, encrypted resources,... update dependencies's version).
| | SonarQube | Primarily for static analysis and code quality, but often integrated with test results for coverage | | ||
| | Jest/JUnit/Pytest| Frameworks for writing and executing tests within codebases | | ||
| | Selenium/Cypress | Frameworks for automated browser-based functional testing | | ||
|
|
There was a problem hiding this comment.
You forgot kubetest (iirc the contributor of testXxx in CDEvents spec) ;-)
The list is a bit of confusion, because it's a mix of test "lib/framework" and workflow tools that can launch those "lib/framework".
Maybe Test should be defined as a step that generates an analysis/report (Test Result), a "non-executable" resource. This resource can be read & used by humans or automates. It also means that those resources should be published/stored (could be ephemeral), as listed in the "Store" section.
| | Code Linting/Formatting | ❌ No | `prettier --check`, `gofmt` | Focuses on code style and consistency, not functional correctness | | ||
| | Dependency Vulnerability Scan| ❌ No | `npm audit`, Snyk scan | Scans for known vulnerabilities in dependencies, not functional test of *your* code | |
There was a problem hiding this comment.
Code linting & Security scan (sometimes coupled, see MegaLinter) are part of the validation process: it's a static analysis of the quality of code (coding convention, best practices, security, typing (for non-compiled language), ...) So it matches the definition above.
It can sometimes be part of the "ops functional". For example, the DevOps & Security teams add policies (admission, network,...) as part of the gatekeeper to deploy into the production Kubernetes cluster, without Linter or Security scan it could be easy to miss this part in the manifest/helm chart, and then be rejected at deployment time instead of during the CI (test & lint).
Why do you want to exclude some kind of "test"?
There was a problem hiding this comment.
We define tests as validating behavioral correctness of code, not style compliance or dependency vulnerabilities. While linting and security scans are important validations, they don't test if your code does what it's supposed to do.
Security scanning, I imagine, will be its own CDEvent under security. Linting itself is static analysis, not behavioral testing. However, when linting is used as a pass/fail validation gate in a pipeline, it could emit a test event. The key is intent and context:
If you're using linting to answer 'does this code meet our quality standards?' → Test event
If you're using linting to transform/fix code → Build event
If you're just generating a report → Neither
CDEvents should reflect how tools are actually used in pipelines, not prescribe rigid categories. A team that treats linting as a critical test gate should be able to emit test events for it.
So to summarize linting, linting itself is NOT a test, but can be used within a test.
| | Artifactory/Nexus| Dedicated artifact repositories for various package types (Maven, npm, Docker, etc.) | | ||
| | Docker Registry | Stores and manages Docker container images | | ||
| | Git/GitHub/GitLab| Version Control Systems that store source code and its history | | ||
| | Amazon S3/Azure Blob Storage/GCS | Object storage services used for archiving logs, backups, or raw data | |
There was a problem hiding this comment.
The list is not closed: artifacts, doc, test report, dataset, ML model, ...
|
Imagine these 2 use cases:
In both CI flows (which could be a single job/task), the only output/outcome at the end is the summary OK/Failed & logs (no published/stored executable, everything was local to the runner or an ephemeral environment, ...), similar to a test result. How do you classify those flows: |
#53 (comment) I think this answers this, @davidB But to be clear
Deploy would cover the execution/running of artifacts in environments, which is distinct from storing them in repositories. We may add Deploy as a future definition. I think. I haven't gotten that far haha. |
3 participants
This commit adds the foundation for definitions to give a common semantics across the various concepts within the SDLC.
This commit explicitly starts with 'build'.
I wanted to first create the page, and once we agree we can figure out where to best link it from