Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add nightly CI testing on RKE2 #726

Closed
mjnagel opened this issue Sep 5, 2024 · 3 comments · Fixed by #808
Closed

Add nightly CI testing on RKE2 #726

mjnagel opened this issue Sep 5, 2024 · 3 comments · Fixed by #808
Assignees
@noahpb
Labels
ci Issues pertaining to CI / Pipelines / Testing enhancement New feature or request nightly Issues pertaining to nightly CI
Milestone
0.30.0

Comments

@mjnagel
Copy link
Contributor

mjnagel commented Sep 5, 2024

Related to #213

Currently we do nightly testing on EKS. In order to expand our testing posture we should also test on RKE2. The EKS testing is contained in these locations:

Definition of done: Nightly test added to spin up RKE2 and deploy core (all flavors, matrixed) on it. This should use external S3 buckets where required. For this initial iteration we should keep it as simple as possible:

  1. Leverage images built by the image builder: https://github.com/defenseunicorns/uds-rke2-image-builder (these are published in the CI AWS account)
  2. Leverage basic terraform to spin up the cluster: Since this is ephemeral we should focus on MVP for now. Potentially could reuse some of the terraform from the image builder.
  3. Use as few AWS dependencies as possible to start. Example: rather than setting up IRSA, just use key based auth for buckets. These may be future enhancements to the RKE2 test but should not be in scope for this initial setup.
@mjnagel mjnagel added enhancement New feature or request ci Issues pertaining to CI / Pipelines / Testing nightly Issues pertaining to nightly CI labels Sep 5, 2024
@mjnagel mjnagel added this to the 0.28.0 milestone Sep 6, 2024
@noahpb noahpb mentioned this issue Sep 23, 2024
Merged
5 tasks
@mjnagel mjnagel modified the milestones: 0.28.0, 0.30.0 Sep 27, 2024
@noahpb
Copy link
Contributor

noahpb commented Oct 1, 2024

Terraform has been validated locally. A multi-node cluster can be deployed with a fresh build of uds-core on top. A mix between the rke2 open source terraform module and IaC from the image builder repository was used. IRSA was found to be necessary due to restrictions in our CI AWS account. Appropriate external infrastructure (such as resources to support OIDC and s3 storage, networking and IAM roles) are also included.

New pipeline is currently pending validation in this PR. Note that the PR includes additional changes to task files and repo structure to support testing across multiple K8s distributions.

@noahpb
Copy link
Contributor

noahpb commented Oct 2, 2024

All CI tests passing, moving this to in review.

@noahpb
Copy link
Contributor

noahpb commented Oct 7, 2024

Currently validating irsa functionality before asking for another review.

@mjnagel mjnagel closed this as completed in c401419 Oct 8, 2024
docandrew pushed a commit that referenced this issue Oct 17, 2024
@noahpb @docandrew
feat: add nightly testing for rke2 (#808)
d6fd452 
## Description
This pull request introduces a new nightly test of uds-core on RKE2. The
pipeline runs in parallel alongside our existing EKS nightly tests.
Included in this PR are additional IaC resources to deploy RKE2, an
additional bundle for RKE2 and updated task files/workflows.

Details:
- Changes `test-infra` directory layout to support multiple k8s
distributions (merges `test-infra/buckets-iac` and `test-infra/rds-iac`
into `test-infra/aws/eks` and adds `test-infra/aws/rke2`)
- Adds IaC for a minimal deployment of RKE2 under `test-infra/aws/rke2`
- Introduces new test bundle for uds-core on RKE2
(`.github/bundles/rke2`), changes `.github/bundles` directory layout to
support additional distros (moves `.github/bundles/*.yaml` to
`.github/bundles/eks`)
- Introduces new workflow for nightly tests of uds-core on rke2
(`.github/workflows/test-rke2.yaml`)
- Updates workflows and task files to support testing across multiple
K8s distros.

Callouts:
- uses existing vpc and subnets in ci account
- the `rke2_version` var will be used to query the latest ami for the
given minor version
- uses rhel amis built from uds-rke2-image-builder repo
- includes support for irsa
- no cluster-autoscaler deployed, currently hardcoded to provision 4
`m5.2xlarge` nodes
- does not deploy any external dependencies apart from s3 buckets for
loki and velero

Example pipeline run with new RKE2 tests can be seen
[here](https://github.com/defenseunicorns/uds-core/actions/runs/11150004170/).

## Related Issue

Fixes #726 

## Type of change

- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Other (security config, docs update, etc)

## Checklist before merging

- [ ] Test, docs, adr added or updated as needed
- [ ] [Contributor
Guide](https://github.com/defenseunicorns/uds-template-capability/blob/main/CONTRIBUTING.md)
followed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@noahpb noahpb

Labels
ci Issues pertaining to CI / Pipelines / Testing enhancement New feature or request nightly Issues pertaining to nightly CI
Projects
None yet
Milestone
0.30.0
Development

Successfully merging a pull request may close this issue.

feat: add nightly testing for rke2 defenseunicorns/uds-core
2 participants
@mjnagel @noahpb