software-design-specification.adoc

Software Design Specification

Table of Contents

• 1. Purpose and Scope
• 2. Definitions and Abbreviations
• 3. References
• 4. Software Design Specification
  • 4.1. Developed components
  • 4.2. Third-party components
• 5. Appendix
• 6. Document History

1. Purpose and Scope

The purpose of this document is to describe the technical realization of the given software system architecture and the software requirements. It states how the design meets the requirements.

2. Definitions and Abbreviations

N/A

3. References

N/A

4. Software Design Specification

4.1. Developed components

As described in the architecture, the system is installed into local namespaces. This document explains the individual components and their interactions.

4.1.1. ODS Pipeline Installation

Build skipping / cache scripts

These scripts may be used by build tasks to skip doing work when no changes have been made compared to a previous task run.

SDS-SHARED-4	cache-build.sh shell script	Caches a build’s outputs and ods artifacts to the build-task cache area. If cache-build is not "true" then exit with error code 0 as in this case nothing should be done. Determine cache location at $ROOT_DIR/.ods-cache/build-task/$CACHE_BUILD_KEY/$git_sha_combined where CACHE_BUILD_KEY is the value of the input parameter below git_sha_combined is made of dash - separated abbreviated git shas for each source directory specified in parameter working-dir followed by those in build-extra-inputs. For each directory the git sha is determined by =$(git rev-parse --short "HEAD:") Copies artifacts to <cache-location>/artifacts Copies build outputs specified in cached-outputs to <cache-location>/output/$i where is is the index of the directory in the colon separated paths of cached-outputs If successful creates file .ods-last-used-stamp in the cache location. Input parameters: working-dir: the sub directory in the repo whose build is cached. cache-build: controls whether build cache is used. build-extra-inputs: list of build source directories (as colon separated string) which influence the build in addition to the files in working-dir. These directories are relative to the repository root. If the contents in these directories change the cache is invalidated so that the build task will rebuild from scratch. cached-outputs: List of build output directories (as colon separated string) to be cached. These directories are relative to the working-dir parameter` cache-build-key: key to distinguish build toolset and build variants build from the same working directory for example for different platforms. cache-location-used-path: specifies path of the tekton task result parameter in which the location of the cache directory used is stored.
SDS-SHARED-5	copy-build-if-cached.sh shell script	Copies build from cache area. If cache-build is not "true" then exit with error code 1 to signal that a build could not be retrieved from the cache (as it is not enabled). Determines cache location at $ROOT_DIR/.ods-cache/build-task/$CACHE_BUILD_KEY/$git_sha_combined in the same way as specified in SDS-SHARED-4. If there is no directory at <cache-location> exit with error code 1 to signal that a build could not be retrieved from the cache. Copy artifacts inside <cache-location>/artifacts to ${ROOT_DIR}/.ods/artifacts Copy output files inside <cache-location>/output to directories specified in cached-outputs. Write the cache-location to file $CACHE_LOCATION_USED_PATH. Touch file .ods-last-used-stamp in the cache location so that the cleanup timestamp is reset. Input parameters: working-dir: the sub directory in the repo whose build is cached. cache-build: controls whether build cache is used. build-extra-inputs: list of build source directories (as colon separated string) which influence the build in addition to the files in working-dir. These directories are relative to the repository root. If the contents in these directories change the cache is invalidated so that the build task will rebuild from scratch. cached-outputs: List of build output directories (as colon separated string) to be cached. These directories are relative to the working-dir parameter` cache-build-key: key to distinguish build toolset and build variants build from the same working directory for example for different platforms. cache-location-used-path: specifies path of the tekton task result parameter in which the location of the cache directory used is stored.

ods-pipeline-start task

SDS-TASK-7	ods-pipeline-start Task resource	Task to start pipeline. References SDS-TASK-8 and executes SDS-TASK-9. Input parameters: TODO
SDS-TASK-8	ods-pipeline-start container image	Container image to start a pipeline. Based on ubi8/ubi-minimal (SDS-EXT-2), includes SDS-EXT-9, SDS-EXT-13, SDS-EXT-27, SDS-EXT-30 and SDS-TASK-9.
SDS-TASK-9	start binary	The task checks out the repository of a given URL and Git ref into the mounted workspace, cleaning previous contents, except for the caching area at ./ods-cache. If the checked out ods.y(a)ml configures any child repositories, those are checked out as well from the configured URL and Git ref (either tag or branch, defaulting to master). All checkouts are by default non-shallow and include submodules. A build task may store cached dependencies under directory .ods-cache/deps/<technology-name>/ where technology-name provides a namespace. For example this could be 'npm' if at some point in the future this would be supported. The task deletes files in folder .ods-cache/deps/. All other files in .ods-cache are reserved for future use. While they are not removed you must not rely on those locations except for experimentation. Context information is stored under .ods for each checked out repository: repository related information: project key, component key, repository name, Git URL, Git (full) ref, Git commit SHA, pull request base and pull request key. OpenShift related information: namespace If the artifact-source parameter is given, any artifacts in the referenced Nexus repository belonging to the same commit being built are downloaded and placed into the respective .ods/artifacts folder of each checked out repository. If any child repository is missing a successful pipeline run artifact for the checked out commit, the task fails. The Bitbucket build status of the commit being built is set to "in progress". The build status links back to the pipeline run.

ods-pipeline-finish task

SDS-TASK-10	ods-pipeline-finish Task resource	Task to finish pipeline. References SDS-TASK-11 and executes SDS-TASK-12. Input parameters: TODO
SDS-TASK-11	ods-pipeline-finish container image	Container image to start a pipeline. Based on ubi8/ubi-minimal (SDS-EXT-2), includes SDS-EXT-30 and SDS-TASK-12.
SDS-TASK-12	finish binary	Sets the Bitbucket build status to "failed" or "successful", depending on whether all tasks succeeded or not. The build status links back to the pipeline run. Creates an artifact for the pipeline run, containing its name and status, provided that all tasks succeeded. If the artifact-target parameter is given, the task uploads all files in any .ods/artifacts folder to the referenced Nexus repository, storing them in a group named /<PROJECT>/<REPOSITORY>/<GIT-COMMIT-SHA>, provided that all tasks succeeded.
SDS-TASK-29	finish binary	Optionally sends a status notification to a webhook receiver. Status notification message, webhook URL, content type, HTTP method, and triggering status values may be configured via a ConfigMap.

Pipeline Manager

SDS-PIPELINE-MANAGER-1	ods-pipeline Service resource	Service (exposing a set of pods) for the pipeline manager
SDS-PIPELINE-MANAGER-2	ods-pipeline Deployment resource	Deployment (providing declarative updates for pods and replica sets) for the pipeline manager. The container template references SDS-PIPELINE-MANAGER-3.
SDS-PIPELINE-MANAGER-3	ods-pipeline-manager container image	Container image to hosting the pipeline manager. Based on ubi8/ubi-minimal (SDS-EXT-2), includes SDS-EXT-30 and SDS-PIPELINE-MANAGER-4.
SDS-PIPELINE-MANAGER-4	pipeline-manager binary	The pipeline manager parses the JSON payload of received Bitbucket webhooks and triggers a pipeline run dependent on the events received. For Git commits of which the commit message instructs skipping CI, no pipelines are triggered. Instructions may be anywhere in the commit message and may be one of (case-insensitive): [ci skip] [skip ci] ***NO_CI*** A pipeline run is created based on the ODS config file read from the Git ref of the repository corresponding to the webhook request. A PVC is created per repository unless it exists already. The name is equal to ods-workspace-<component> (shortened to 63 characters if longer). This PVC is then used in the pipeline run as a shared workspace. When no other pipeline run for the same repository is running or pending, the created pipeline run starts immediately. Otherwise a pending pipeline run is created, and a periodic polling is kicked off to allow the run to start once possible. Since the pipeline manager does not persist state about pending pipeline runs, polling is also started for all repositories in the related Bitbucket project when the server boots. Pipelines runs are pruned when a webhook trigger is received. Pipeline runs that are newer than the configured time window are protected from pruning. Older pipeline runs are cleaned up to not grow beyond the configured maximum amount. The pruning strategy is applied per repository.

Artifact Download

SDS-DLD-1

artifact-download binary

The binary receives flags from the user identifying: OpenShift namespace Git repository (project/repository) Git tag The OpenShift namespace is used to retrieve configuration and secrets required to communicate with Bitbucket and Nexus. The ods.yaml of the Git repository is retrieved at given Git tag to detect any subrepositories. If the given tag is WIP, the repository information is not retrieved from Bitbucket but located from the .git directory in the working directory. For all repositories in scope, the artifacts in the corresponding groups in Nexus are downloaded to the local host. The files are placed into artifacts-out/<TAG> (customizable via --output).

Installation / Update

SDS-SETUP-1	Helm chart ods-pipeline	The Helm chart contains resources related to the pipeline manager, as well as config maps, secrets and tasks supporting pipeline runs.
SDS-SETUP-2	web-terminal-install.sh script	The script is supposed to be downloaded and piped into bash. The script installs the prerequisites not already present in the Web Terminal image (Helm plugin helm-diff), clones the Git repository and then runs ./install.sh.
SDS-SETUP-3	install.sh script	The script installs the Helm chart located in deploy/chart. Further, it: creates the pipeline serviceaccount if it does not exist already creates secrets holding relevant credentials (e.g. Bitbucket access token), either by prompting the user for values, or taking input from command line flags in case of an update, modifies existing secrets when command line flags are given adds the tekton.dev/git-0 annotation to the ods-bitbucket-auth secret (pointing to the Bitbucket URL) and associate the secret with the pipeline serviceaccount to enable git clone` in the ods-pipeline-start task

4.2. Third-party components

ID	Name	Version	Description	Link
SDS-EXT-2	Red Hat Universal Base Image 8 Minimal	8.4	Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. It is maintained by Red Hat and updated regularly.	https://catalog.redhat.com/software/containers/ubi8/ubi-minimal/5c359a62bed8bd75a2c3fba8
SDS-EXT-9	Git	2.39	Distributed version control system.	https://git-scm.com
SDS-EXT-13	openssh-clients	8.0	Clients necessary to make encrypted connections to SSH servers.	https://www.openssh.com
SDS-EXT-27	Git LFS	3.0.2	Git Large File Storage extension for versioning large files.	https://git-lfs.github.com/

5. Appendix

N/A

6. Document History

As this document is version controlled in Git, all changes are tracked as commits. The history of changes to this file can be retrieved via git log --oneline --no-merges docs/design/software-design-specification.adoc.