relationship-shared-library.adoc

ODS Pipeline vs. ODS Jenkins Shared Library

The ODS Jenkins Shared library provides pipelines and tasks to support CI/CD flows using Jenkins. ODS Pipeline is an alternative approach using Tekton instead of Jenkins. Therefore, on a high-level, both ODS Pipeline and ODS Jenkins Shared library share the same goals. However, there are many differences between the two options, not only in terms of functionality and maturity, but also in terms of approach and concepts.

High-Level Comparison

The Jenkins shared library can be used in a Jenkinsfile to avoid repeatedly creating common tasks such as building a container image from a Dockerfile in the Git repository.

In the approach chosen by ods-pipeline, the pipeline definition in the ods.y(a)ml file is a bit like the Jenkinsfile: it defines which steps happen in the CI pipeline. The Tekton tasks (such as ods-pipeline-helm-deploy) are a bit like the stages provided by the shared library (such as odsComponentPipelineRolloutOpenShiftDeployment). The main difference between the Jenkins shared library and the Tekton-based approach is that users of Jenkins can script their CI pipeline in the Jenkinsfile, whereas Tekton pipelines are only a series of tasks defined in YAML, which is way less flexible. Another important difference is that the Jenkins shared library does not contain any language-specific instructions (such as how to build Java applications or how to build Python applications), in contrast to ods-pipeline, which provides one (opinionated) companion task per language (e.g. ods-pipeline-gradle-build).

Release Manager

The concept of multiple component repos and one umbrella repo is the same in both options. With ods-pipeline, each individual component repo has an ods.yml which defines a pipeline. This pipeline produces artifacts, stored in Nexus, which can be used later by the umbrella repo.

The umbrella repo also has an ods.yml defining a pipeline. In contrast to the Jenkins approach, the umbrella repo (release manager) does not execute any pipelines from child repos though. Instead, the umbrella repo simply defines the tasks it wants to execute. Those tasks can make use of the artifacts created in earlier component pipeline runs.

The orchestration pipeline currently has fixed stages (build, deploy, test, release, finalise) in which different work is done for each sub repo, depending on its type (for example, for normal ODS components the build stage calls the Jenkinsfile, and for test ODS components the test stage calls the Jenkinsfile).

Further, each stage in the orchestration pipeline creates documents, sometimes at the beginning of the stage, sometimes before each repo, sometimes after each repo and sometimes at the end of the stage.

The Tekton pipeline takes a different approach: all such functionality would be provided by one or two tasks. These tasks would download information from Jira, get the artifacts from the subrepos, and then assemble the full set of documents. The tasks would be part of a regular pipeline.

Following are some details on what that means practically:

• Reuse of the existing Groovy code would be possible, at least partially. All the wiring (the pipeline and the stages) would need to be refactored, but the services and util classes should be reusable as is)

• Deployment could be handled by the ods-pipeline-helm-deploy task. In the shared lib, there is a huge overlap between component deploy and orchestration deploy. In the new world, we might be able to merge this into one task. The ods-pipeline-helm-deploy task already supports collecting dependency charts from subrepos and pushing images to other namespaces. Some additional features from the orchestration pipeline would need to be ported (such as collecting IP address of the deployed pod and storing that as an artifact)

• Tests (e.g. Geb/Spock) obviously need to be run in the "umbrella pipeline run" …​ in this case, using artifacts from previous runs is not possible. Instead, this would be achieved by explicitly adding the task ref of the "test task" in the umbrella repos ods.yml.

• The AWS quickstarter also needs to do work during the "umbrella pipeline run", and again the way to solve this is to have the task ref in the umbrella repos ods.yml. A pattern emerges: either you can reuse the artifacts instead of running the tasks (e.g. building container images) or you need to add the task into the umbrella repos ods.yml.

Finally, a few thoughts on enforcing certain tasks / flexibility / adhering to a standard. This is in the context of GAMP or medical device software, in which a platform would like to ensure certain things to happen before software reaches production.

In general, we see two main approaches to address this:

1. Introduce next to the pipeline field in ods.yml another field which describes a certain kind of pipeline in which some tasks are predefined and cannot be changed by the user. E.g. a samdPipeline or levaPipeline could pre-configure a pipeline with certain tasks and allow only very limited customisation.

2. Use the "release manager / doc gen task" described above to verify certain tasks have run or that certain artifacts are present. E.g. the task could check that there are xUnit test results for all components which are part of the app. Further, the task could check that all tasks of the pipeline are of kind Task and start with ods-`. Those tasks can be assumed to be qualified. If the user added other tasks, they would need to explicitly opt-in as they need to provide their own qualification documents for those tasks then.

Document generation

At the moment, the orchestration pipeline only sends requests to the doc gen service, which is running in each namespace, and receives rendered documents. We believe that this architecture is needlessly complex for the Tekton approach. Instead of having a long-running service in each namespace which consumes resources and is hard to maintain (as every initiative needs to do that themselves), we could simply call the service classes from the doc gen services directly from the Groovy code of the new "release manager / doc gen task".

Quickstarters

Quickstarters currently provide a Jenkinsfile.template which will get rendered into the resulting component repository. With ods-pipeline, this would be replaced by an ods.yml file.

Further, the quickstarter provisioning itself is also done by a Jenkins pipeline, based on the Jenkinsfile in the quickstarter. This also needs to be replaced with something. Right now, the quickstart process consists of creating a repo, and creating OpenShift resources. We would stop creating OpenShift resources in the new world, and simply require each quickstarter to supply a Helm chart that is then deployed automatically with the first pipeline run (we need to be infrastructure-as-code with Helm as no magic export functionality exists like with Tailor). With that in mind, we only need to create a repo. Often, that step includes running tech-specific tasks (e.g. generators). We suggest that quickstarter authors define a TaskRun, which then gets executed. That way they can pick their own container image and do whatever they want, in whatever language they want.

Users will still be able to consume quickstarters via the provisioning app.

Authoring a new quickstarter means potentially creating a new task (say a Rust task). We do not know yet how a less official task could be shared easily …​ each namespace would need to install that task separately. To create such a task, authors need to have:

• expert knowledge of the technology involved in the QS (such as Rust for example)

• some knowledge of Helm to create the chart will be required

• some knowledge of Kubernetes and Tekton to write the task definition

• enough knowledge of Go to supply an automated test using the test framework

• some knowledge of any programming language to write the logic of the task. Go would be preferred here but is not required. If it is very little logic, bash will do fine as well.

Pro’s / Con’s and Limitations

The following pro’s and con’s are mostly from a platform perspective, not necessarily from an individual developer’s perspective. Also, these are in relation to the existing shared library approach.

Pro

• Tekton is very rigid compared to Jenkins: the task list is a simple array. No crazy scripting possible.

• Tekton tasks have a pretty clear interface (parameters and results). Updating between versions should be easy and predictable. However, we need to be cautious not to depend too much on workspace state.

• No long-running Jenkins instance which saves 5Gi memory per project

• No complicated base images - the existing Jenkins solution is a bit brittle where many updates of the base images (be it Jenkins, Java, or something else) broke something down the road

• Jenkins had only one agent image, which made it hard to use for monorepos using multiple technologies (e.g. TypeScript and Java). The Tekton approach should handle monorepos and multiple repos equally well.

• The Tekton implementation can run in a pure Kubernetes cluster, allowing the test suite to be executed in GitHub Actions. Also, the target cluster does not have to be OpenShift, allowing to deploy into EKS for example.

• The artifact approach avoids the need to run all components in the release manager pipeline (speeding things up) while at the same time storing all relevant information which should be useful for GxP/SaMD.

• Possibility to run in and deploy to Kubernetes as well as OpenShift

• Support for Helm instead of Tailor

• Third party dependency caching (only used in Go task so far) and build skipping in multi-repo builds (only in PR state so far)

• Easy notification of pipeline status in MS Teams channel

Con

• Tekton tasks are only customizable via parameters so many people might need to create their own tasks because the platform cannot cover all use cases.

• Each task is one pod - this causes performance overhead as spinning up pods is a bit slow. Jenkins spins up only one pod and therefore is faster.

• There is no way to install plugins or use the UI to e.g. see test execution trends.

Limitations

• OpenShift Pipelines does not provide a way to override task resources from a Pipeline resource. Tekton has added support for this in PipelineRun and TaskRun resources recently, but this new version is not picked up by OpenShift Pipelines yet. ODS Pipeline provides a workaround by allowing task resources to be defined via the Helm chart.

• Tekton does not provide a way to specify sidecars from a Pipeline resource. ODS Pipeline provides a workaround by allowing task sidecards to be defined via the Helm chart.

• As tasks are pods, one needs a PVC to work on. Using a PVC has an effect on how many pipelines can run in parallel. At the moment, ODS Pipeline supports one PVC per repo. This prevents parallel runs for different branches.

Key Learnings from the Jenkins shared library

• Do not build components in parallel. The Jenkins release manager builds all components in parallel to assemble a promotable package. This is very challenging in terms of quotas.

• Do not commit into the repository being built. This leads may lead to another run being triggered and should be avoided, plus it requires being able to push into the repo.

• Do not export OpenShift resources on-the-fly. This is error prone, non-traceable and potentially even dangerous. Resources must be specified explicitly in the repository upfront.

• Do not control too much. Use standard functionality and do not add to many checks / assumptions on top. Otherwise developers are frustrated because of all the magic and limitations imposed, which may not be relevant for their use case.

• Testing must be supported really well. Tasks must be testable in isolation, locally, and automated on GitHub. It is not enough to mock everything. Tests must be possible on different levels of integration (e.g. against real SQ instances)