Skip to content

Commit b0f45da

Browse files
benjaminjbbenjaminjb
authored
envsubst for KUTTL (CrunchyData#2984)
Add a /source/ folder with example tests (pulled from e2e folder); Add make targets used (1) to create generated KUTTL tests and (2) to create & run generated KUTTL tests Add notes to testing/kuttl/readme.md Issue [sc-13318]
1 parent 54a34a4 commit b0f45da

20 files changed

+209
-10
lines changed

‎.gitignore

+1
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
11
.DS_Store
22
/vendor/
33
tools
4+
/testing/kuttl/generated/

‎Makefile

+30
Original file line numberDiff line numberDiff line change
@@ -217,6 +217,35 @@ check-kuttl:
217217
${PGO_KUBE_CLIENT} ${KUTTL_TEST} \
218218
--config testing/kuttl/kuttl-test.yaml
219219

220+
# Set reasonable default for KUTTL_ env vars for templating
221+
KUTTL_PG_VERSION ?= 14
222+
KUTTL_PG_UPGRADE_FROM_VERSION ?= 13
223+
KUTTL_PSQL_IMAGE ?= registry.developers.crunchydata.com/crunchydata/crunchy-postgres:centos8-14.1-0
224+
225+
# TODO: When I attempted to loop through the directories and find the yaml files,
226+
# due to how makefile expands variables, each file would end up in each directory.
227+
# So this implementation loops through to create the folders under generated
228+
# and then loops through all the yaml files, using a `sed` substition to place them
229+
# For safety, this removes the `testing/kuttl/generated` folder if it exists
230+
.PHONY: generate-kuttl
231+
generate-kuttl:
232+
[ ! -d testing/kuttl/generated ] || rm -r testing/kuttl/generated
233+
for SRCDIR in $(shell ls testing/kuttl/source); do \
234+
mkdir -p testing/kuttl/generated/$${SRCDIR}; \
235+
done;
236+
for FILEPATH in $(wildcard testing/kuttl/source/*/*.yaml); do \
237+
NEW_FILEPATH=`echo $${FILEPATH} | sed 's/source/generated/'`; \
238+
KUTTL_PG_VERSION=$(KUTTL_PG_VERSION) \
239+
KUTTL_PG_UPGRADE_FROM_VERSION=$(KUTTL_PG_UPGRADE_FROM_VERSION) \
240+
KUTTL_PSQL_IMAGE=$(KUTTL_PSQL_IMAGE) \
241+
envsubst < $${FILEPATH} > $${NEW_FILEPATH}; \
242+
done
243+
244+
.PHONY: check-generated-kuttl
245+
check-generated-kuttl: generate-kuttl
246+
${PGO_KUBE_CLIENT} ${KUTTL_TEST} \
247+
--config testing/kuttl/kuttl-test-generated.yaml
248+
220249
.PHONY: check-generate
221250
check-generate: generate-crd generate-deepcopy generate-rbac
222251
git diff --exit-code -- config/crd
@@ -226,6 +255,7 @@ check-generate: generate-crd generate-deepcopy generate-rbac
226255
clean: clean-deprecated
227256
rm -f bin/postgres-operator
228257
rm -f config/rbac/role.yaml
258+
[ ! -d testing/kuttl/generated ] || rm -r testing/kuttl/generated
229259
[ ! -d build/crd/generated ] || rm -r build/crd/generated
230260
[ ! -d hack/tools/envtest ] || rm -r hack/tools/envtest
231261
[ ! -n "$$(ls hack/tools)" ] || rm hack/tools/*

‎testing/kuttl/README.md

+33-1
Original file line numberDiff line numberDiff line change
@@ -53,4 +53,36 @@ the files would be named
5353
```console
5454
00--cluster.yaml # note the extra `-` to ensure that it sorts above the following file
5555
00-assert.yaml
56-
```
56+
```
57+
58+
### Generating tests
59+
60+
KUTTL is good at setting up K8s objects for testing, but does not have a native way to dynamically
61+
change those K8s objects before applying them. That means that, if we wanted to write a cluster
62+
connection test for PG 13 and PG 14, we would end up writing two nearly identical tests.
63+
64+
Rather than write those multiple tests, we are using `envsubst` to replace some common variables
65+
in a `source` template YAML and writing those files to the `testing/kuttl/generated` folder.
66+
67+
These templated test files can be generated by setting some variables in the command line and
68+
calling the `make generate-kuttl` target:
69+
70+
```console
71+
KUTTL_PG_VERSION=13 KUTTL_PSQL_IMAGE=registry.developers.crunchydata.com/crunchydata/crunchy-postgres:centos8-13.5-0 make generate-kuttl
72+
```
73+
74+
This will loop through the folders in the `source` folder and create matching folders in the
75+
`generated` folder that can be checked for correctness before running the tests. (The files in
76+
the `generated` folder will not be checked into github; our CI runn will generate and test the
77+
files from scratch.)
78+
79+
For ease of use, the `check-generated-kuttl` make target will both generate those tests and test
80+
them, again with the set environment variables:
81+
82+
```console
83+
KUTTL_PG_VERSION=13 KUTTL_PSQL_IMAGE=registry.developers.crunchydata.com/crunchydata/crunchy-postgres:centos8-13.5-0 make check-generated-kuttl
84+
```
85+
86+
To prevent errors, we want to set defaults for all the environment variables used in the `source`
87+
YAML files; so if you add a new test with a new variable, please update the Makefile with a
88+
reasonable/preferred default.

‎testing/kuttl/e2e/pgbackrest-init/README.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -3,4 +3,4 @@
33
* 00: Create a cluster with two PVC repos and set up for manual backups to go to the second; verify that the PVCs exist and that the backup job completed successfully
44
* 01: Run pgbackrest-initialization.sh, which checks that the status matches the expected status of `mixed` (because the second repo in the repo list has not yet been pushed to) and that there is only one full backup
55
* 02: Use `kubectl` to annotate the cluster to initiate a manual backup; verify that the job completed successfully
6-
* 03: Rerun pgbackrest-initialization.sh, now expecting the status to be `ok` since both repos have been pushed to and there to be two full backups
6+
* 03: Rerun pgbackrest-initialization.sh, now expecting the status to be `ok` since both repos have been pushed to and there to be two full backups

‎testing/kuttl/kuttl-test-generated.yaml

+14
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
apiVersion: kuttl.dev/v1beta1
2+
kind: TestSuite
3+
testDirs:
4+
- testing/kuttl/generated/
5+
timeout: 180
6+
parallel: 2
7+
# by default kuttl will run in a generated namespace to override
8+
# that functionality simply uncomment the line below and replace
9+
# postgres-operator with the desired namespace to run in.
10+
# namespace: postgres-operator
11+
# By default kuttl deletes the resources created during a test.
12+
# For debugging, it may be helpful to uncomment the following line
13+
# in order to inspect the resources.
14+
# skipDelete: true

‎testing/kuttl/source/cluster-start/00--cluster.yaml

+25
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
apiVersion: postgres-operator.crunchydata.com/v1beta1
2+
kind: PostgresCluster
3+
metadata:
4+
name: cluster-start
5+
spec:
6+
postgresVersion: ${KUTTL_PG_VERSION}
7+
instances:
8+
- name: instance1
9+
dataVolumeClaimSpec:
10+
accessModes:
11+
- "ReadWriteOnce"
12+
resources:
13+
requests:
14+
storage: 1Gi
15+
backups:
16+
pgbackrest:
17+
repos:
18+
- name: repo1
19+
volume:
20+
volumeClaimSpec:
21+
accessModes:
22+
- "ReadWriteOnce"
23+
resources:
24+
requests:
25+
storage: 1Gi

‎testing/kuttl/e2e/cluster-start/00-assert.yaml renamed to ‎testing/kuttl/source/cluster-start/00-assert.yaml

File renamed without changes.

‎testing/kuttl/source/cluster-start/01--psql-connect.yaml

+28
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
apiVersion: batch/v1
2+
kind: Job
3+
metadata:
4+
name: psql-connect
5+
spec:
6+
template:
7+
spec:
8+
restartPolicy: "OnFailure"
9+
containers:
10+
- name: psql
11+
image: ${KUTTL_PSQL_IMAGE}
12+
command:
13+
- psql
14+
- -c
15+
- "select version();"
16+
env:
17+
- name: PGHOST
18+
valueFrom: { secretKeyRef: { name: cluster-start-pguser-cluster-start, key: host } }
19+
- name: PGPORT
20+
valueFrom: { secretKeyRef: { name: cluster-start-pguser-cluster-start, key: port } }
21+
- name: PGDATABASE
22+
valueFrom: { secretKeyRef: { name: cluster-start-pguser-cluster-start, key: dbname } }
23+
- name: PGUSER
24+
valueFrom: { secretKeyRef: { name: cluster-start-pguser-cluster-start, key: user } }
25+
- name: PGPASSWORD
26+
valueFrom: { secretKeyRef: { name: cluster-start-pguser-cluster-start, key: password } }
27+
# Do not wait indefinitely.
28+
- { name: PGCONNECT_TIMEOUT, value: '5' }

‎testing/kuttl/e2e/cluster-start/01-assert.yaml renamed to ‎testing/kuttl/source/cluster-start/01-assert.yaml

File renamed without changes.

‎testing/kuttl/e2e/major-upgrade/00--cluster.yaml renamed to ‎testing/kuttl/source/major-upgrade/00--cluster.yaml

+1-1
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ kind: PostgresCluster
33
metadata:
44
name: major-upgrade
55
spec:
6-
postgresVersion: 13
6+
postgresVersion: ${KUTTL_PG_UPGRADE_FROM_VERSION}
77
instances:
88
- name: instance1
99
dataVolumeClaimSpec:

‎testing/kuttl/e2e/major-upgrade/00-assert.yaml renamed to ‎testing/kuttl/source/major-upgrade/00-assert.yaml

+1-1
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ kind: PostgresCluster
33
metadata:
44
name: major-upgrade
55
spec:
6-
postgresVersion: 13
6+
postgresVersion: ${KUTTL_PG_UPGRADE_FROM_VERSION}
77
status:
88
instances:
99
- name: instance1

‎testing/kuttl/e2e/major-upgrade/01-postgres-connect.yaml renamed to ‎testing/kuttl/source/major-upgrade/01-postgres-connect.yaml

+1-1
Original file line numberDiff line numberDiff line change
@@ -2,4 +2,4 @@ apiVersion: kuttl.dev/v1beta1
22
kind: TestStep
33
commands:
44
# When the cluster comes up, verify that the Postgres major version is correct
5-
- script: CLUSTER=major-upgrade ../../scripts/postgres-version-check.sh "13"
5+
- script: CLUSTER=major-upgrade ../../scripts/postgres-version-check.sh "${KUTTL_PG_UPGRADE_FROM_VERSION}"

‎testing/kuttl/e2e/major-upgrade/02--cluster.yaml renamed to ‎testing/kuttl/source/major-upgrade/02--cluster.yaml

+2-2
Original file line numberDiff line numberDiff line change
@@ -5,8 +5,8 @@ metadata:
55
spec:
66
upgrade:
77
enabled: true
8-
fromPostgresVersion: 13
9-
postgresVersion: 14
8+
fromPostgresVersion: ${KUTTL_PG_UPGRADE_FROM_VERSION}
9+
postgresVersion: ${KUTTL_PG_VERSION}
1010
instances:
1111
- name: instance1
1212
dataVolumeClaimSpec:

‎testing/kuttl/e2e/major-upgrade/02-assert.yaml renamed to ‎testing/kuttl/source/major-upgrade/02-assert.yaml

+1-1
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ kind: PostgresCluster
33
metadata:
44
name: major-upgrade
55
spec:
6-
postgresVersion: 14
6+
postgresVersion: ${KUTTL_PG_VERSION}
77
status:
88
instances:
99
- name: instance1

‎testing/kuttl/e2e/major-upgrade/03-postgres-connect.yaml renamed to ‎testing/kuttl/source/major-upgrade/03-postgres-connect.yaml

+1-1
Original file line numberDiff line numberDiff line change
@@ -2,4 +2,4 @@ apiVersion: kuttl.dev/v1beta1
22
kind: TestStep
33
commands:
44
# When the cluster comes back up, check that the Postgres major version has changed
5-
- script: CLUSTER=major-upgrade ../../scripts/postgres-version-check.sh "14"
5+
- script: CLUSTER=major-upgrade ../../scripts/postgres-version-check.sh "${KUTTL_PG_VERSION}"

‎testing/kuttl/e2e/major-upgrade/README.md renamed to ‎testing/kuttl/source/major-upgrade/README.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -2,4 +2,4 @@
22
* 00: Create a standard Postgres 13 cluster; verify that the cluster instance is created as expected and that the initial backup job completed successfully.
33
* 01: Run postgres-version-check.sh, which verifies the expected Postgres version is returned.
44
* 02: Update the spec from step 00 to upgrade from 13 to 14; verify that the instance is ready, the version on the spec is correct, and both the backup and upgrade Jobs have completed successfully.
5-
* 03: Rerun postgres-version-check.sh to verify the expected Postgres version is returned.
5+
* 03: Rerun postgres-version-check.sh to verify the expected Postgres version is returned.

‎testing/kuttl/source/replica-read/00--cluster.yaml

+26
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
apiVersion: postgres-operator.crunchydata.com/v1beta1
2+
kind: PostgresCluster
3+
metadata:
4+
name: replica-read
5+
spec:
6+
postgresVersion: ${KUTTL_PG_VERSION}
7+
instances:
8+
- name: instance1
9+
dataVolumeClaimSpec:
10+
accessModes:
11+
- "ReadWriteOnce"
12+
resources:
13+
requests:
14+
storage: 1Gi
15+
replicas: 2
16+
backups:
17+
pgbackrest:
18+
repos:
19+
- name: repo1
20+
volume:
21+
volumeClaimSpec:
22+
accessModes:
23+
- "ReadWriteOnce"
24+
resources:
25+
requests:
26+
storage: 1Gi

‎testing/kuttl/e2e/replica-read/00-assert.yaml renamed to ‎testing/kuttl/source/replica-read/00-assert.yaml

File renamed without changes.

‎testing/kuttl/source/replica-read/01--psql-replica-read.yaml

+43
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
apiVersion: batch/v1
2+
kind: Job
3+
metadata:
4+
name: psql-replica-read
5+
spec:
6+
backoffLimit: 3
7+
template:
8+
spec:
9+
restartPolicy: "OnFailure"
10+
containers:
11+
- name: psql
12+
image: ${KUTTL_PSQL_IMAGE}
13+
command:
14+
# https://www.postgresql.org/docs/current/plpgsql-errors-and-messages.html#PLPGSQL-STATEMENTS-ASSERT
15+
# If run on a non-replica, this assertion fails, resulting in the pod erroring
16+
# Note: the `$$$$` is reduced to `$$` by Kubernetes.
17+
# - https://kubernetes.io/docs/tasks/inject-data-application/
18+
- psql
19+
- -qc
20+
- |
21+
DO $$$$
22+
BEGIN
23+
ASSERT pg_is_in_recovery();
24+
END $$$$;
25+
env:
26+
# The Replica svc is not held in the user secret, so we hard-code the Service address
27+
# (using the downstream API for the namespace)
28+
- name: NAMESPACE
29+
valueFrom:
30+
fieldRef:
31+
fieldPath: metadata.namespace
32+
- name: PGHOST
33+
value: "replica-read-replicas.$(NAMESPACE).svc"
34+
- name: PGPORT
35+
valueFrom: { secretKeyRef: { name: replica-read-pguser-replica-read, key: port } }
36+
- name: PGDATABASE
37+
valueFrom: { secretKeyRef: { name: replica-read-pguser-replica-read, key: dbname } }
38+
- name: PGUSER
39+
valueFrom: { secretKeyRef: { name: replica-read-pguser-replica-read, key: user } }
40+
- name: PGPASSWORD
41+
valueFrom: { secretKeyRef: { name: replica-read-pguser-replica-read, key: password } }
42+
# Do not wait indefinitely.
43+
- { name: PGCONNECT_TIMEOUT, value: '5' }

‎testing/kuttl/e2e/replica-read/01-assert.yaml renamed to ‎testing/kuttl/source/replica-read/01-assert.yaml

File renamed without changes.

0 commit comments

Comments
 (0)