Replace kind with k3d (#849)

When running some tests I was able to reduce the time they take to run
quite a bit by using k3d rather than kind.

In smaller tests, this was significant, for example the smoketests ran
in less than half the time. On larger tests, it shaves a couple of
minutes off (on the ~10 I ran anyways). Pressumably this is just the
difference in the time it takes to start/stop k3d vs kind.

To make this all work, I had to create a custom kubetest2 deployer for
k3d. It's in _e2e/kubetest2-k3d_ and largely is a copy of the kind
deployer updated to use k3d instead.

Author: pseudomuto

.github/workflows/e2e-kind-create.yaml

@@ -37,5 +37,5 @@ jobs:
           path: "~/.cache/bazel"
           key: bazel
 
-      - name: End-to-end (kind)
-        run: make test/e2e/kind-create
+      - name: End-to-end (k3d)
+        run: make test/e2e/k3d-create

.github/workflows/e2e-kind-decommission.yaml

@@ -37,5 +37,5 @@ jobs:
           path: "~/.cache/bazel"
           key: bazel
 
-      - name: End-to-end (kind)
-        run: make test/e2e/kind-decommission
+      - name: End-to-end (k3d)
+        run: make test/e2e/k3d-decommission

.github/workflows/e2e-kind-upgrades.yaml

@@ -37,5 +37,5 @@ jobs:
           path: "~/.cache/bazel"
           key: bazel
 
-      - name: End-to-end (kind)
-        run: make test/e2e/kind-upgrades
+      - name: End-to-end (k3d)
+        run: make test/e2e/k3d-upgrades

.github/workflows/e2e-kind-upgradessha256.yaml

@@ -37,5 +37,5 @@ jobs:
           path: "~/.cache/bazel"
           key: bazel
 
-      - name: End-to-end (kind)
-        run: make test/e2e/kind-upgradessha256
+      - name: End-to-end (k3d)
+        run: make test/e2e/k3d-upgradessha256

.github/workflows/e2e-kind-versionchecker.yaml

@@ -38,4 +38,4 @@ jobs:
           key: bazel
 
       - name: End-to-end (kind)
-        run: make test/e2e/kind-versionchecker
+        run: make test/e2e/k3d-versionchecker

.github/workflows/nightly-smoketest.yaml

@@ -31,7 +31,7 @@ jobs:
     strategy:
       matrix:
         # supported Kubenetes versions
-        NODE_VERSION: [1.18.19, 1.19.11, 1.20.7, 1.21.2, 1.22.1]
+        NODE_VERSION: [1.19.16, 1.20.15, 1.21.9, 1.22.6, 1.23.3]
     steps:
       - uses: actions/checkout@v3
       - name: Mount bazel cache

DEVELOPER.md

@@ -6,14 +6,12 @@ See: https://docs.bazel.build/versions/master/install.html
 
 Take a look at the [Makefile](https://github.com/cockroachdb/cockroach-operator/blob/master/Makefile) for various
 targets you can run. The e2e testing requires a container engine like docker running as it starts a K8s cluster with
-Kind. The Makefile is simply a wrapper for bazel commands and does not have any build functionality in it.
+[k3d]. The Makefile is simply a wrapper for bazel commands and does not have any build functionality in it.
 
 You will also need kubectl and a running Kubernetes cluster to run the k8s targets like `make k8s/apply`.
 
 Bazel has been primarily tested on Linux but "should" run on macOS as well. Windows at this time has not been tested.
 
-TODO: notes on python configuration for the host.
-
 If you are new to Bazel take a look at the Workspace file, as it lays out which rule sets we are using, such as docker
 rules and k8s rules. The Makefile notes below talk about the different target groups that run various bazel commands.
 
@@ -39,25 +37,24 @@ Please run the testing targets locally before you push a PR.
 
 ## Running Locally
 
-We use [kind] to run the operator in a local environment. This can provide a faster feedback cycle while developing
+We use [k3d] to run the operator in a local environment. This can provide a faster feedback cycle while developing
 since there's no need to set up a remote GKE/OpenShift cluster.
 
-[kind]: https://kind.sigs.k8s.io/
+[k3d]: https://k3d.io
 
 **make dev/up**
 
 This command will get everything set up for you to begin testing out the operator. Specifically it will:
 
-* Start a local docker registry (via docker run) and configure it to work with kind/K8s
-* Start a kind cluster named test (context=kind-test)
+* Start a k3d cluster named test (context=k3d-test) with a managed docker registry
 * Install the CRDs
-* Build a docker image and publish it to the local registry
+* Build a docker image and publish it to the k3d registry
 * Deploy the operator and wait for it to be available
 * Ensure your shell is configured to use the new cluster (kube context)
 
 **make dev/down**
 
-Tears down the kind cluster.
+Tears down the k3d cluster.
 
 ## Testing CR Database
 
@@ -108,8 +105,8 @@ When you have removed your example and the persitent volumes you can use the fol
 1. If you do not have a GKE cluster we have a helper script `./hack/create-gke-cluster.sh -c test`.
 1. Locate the latest released tag here  https://github.com/cockroachdb/cockroach-operator/tags
 1. Clone the tag `git clone --depth 1 --branch <tag_name> https://github.com/cockroachdb/cockroach-operator`
-1. Execute `kubectl apply -f config/crd/bases/crdb.cockroachlabs.com_crdbclusters.yaml`
-1. Execute `kubectl apply -f manifests/operator.yaml`
+1. Execute `kubectl apply -f install/crds.yaml`
+1. Execute `kubectl apply -f install/operator.yaml`
 1. Check that the operator has deployed properly
 
 The examples directory contains various examples, for example you can run `kubectl apply -f examples/example.yaml`.

Makefile

@@ -60,7 +60,7 @@ test/verify:
 test/lint:
 	bazel run //hack:verify-gofmt
 
-# NODE_VERSION refers the to version of the kindest/node image. E.g. 1.22.1
+# NODE_VERSION refers the to patch version of Kubernetes (e.g. 1.22.6)
 .PHONY: test/smoketest
 test/smoketest:
 	@bazel run //hack/smoketest -- -dir $(PWD) -version $(NODE_VERSION)
@@ -71,10 +71,9 @@ test/smoketest:
 test/e2e-short:
 	bazel test //e2e/... --test_arg=--test.short
 
-#
 # End to end testing targets
 #
-# kind: use make test/e2e/kind
+# k3d: use make test/e2e/k3d
 # gke: use make test/e2e/gke
 #
 # kubetest2 binaries from the kubernetes testing team is used
@@ -83,31 +82,32 @@ test/e2e-short:
 # Once the repo releases binaries we should vendor the tag or
 # download the built binaries.
 
-# This target is used by kubetest2-tester-exec when running a kind test
-# This target exportis the kubeconfig from kind and then runs
-# k8s:k8s -type kind which checks to see if kind is up and running.
+# This target is used by kubetest2-tester-exec when running a k3d test
+# It run k8s:k8s -type k3d which checks to see if k3d is up and running.
 # Then bazel e2e testing is run.
-# An example of calling this is using make test/e2e/testrunner-kind-upgrades
-test/e2e/testrunner-kind-%: PACKAGE=$*
-test/e2e/testrunner-kind-%:
-	bazel-bin/hack/bin/kind export kubeconfig --name $(CLUSTER_NAME)
-	bazel run //hack/k8s:k8s -- -type kind
+# An example of calling this is using make test/e2e/testrunner-k3d-upgrades
+test/e2e/testrunner-k3d-%: PACKAGE=$*
+test/e2e/testrunner-k3d-%:
+	bazel run //hack/k8s:k8s -- -type k3d
 	bazel test --stamp //e2e/$(PACKAGE)/... --test_arg=-test.v --test_arg=-test.parallel=8 --test_arg=parallel=true
 
-# Use this target to run e2e tests using a kind k8s cluster.
-# This target uses kind to start a k8s cluster  and runs the e2e tests
+# Use this target to run e2e tests using a k3d k8s cluster.
+# This target uses k3d to start a k8s cluster  and runs the e2e tests
 # against that cluster.
-# This is the main entrypoint for running the e2e tests on kind.
-# This target runs kubetest2 kind that starts a kind cluster
+#
+# This is the main entrypoint for running the e2e tests on k3d.
+# This target runs kubetest2 k3d that starts a k3d cluster
 # Then kubetest2 tester exec is run which runs the make target
-# test/e2e/testrunner-kind.
+# test/e2e/testrunner-k3d.
 # After the tests run the cluster is deleted.
 # If you need a unique cluster name override CLUSTER_NAME.
-test/e2e/kind-%: PACKAGE=$*
-test/e2e/kind-%:
-	bazel build //hack/bin/...
-	PATH=${PATH}:bazel-bin/hack/bin kubetest2 kind --cluster-name=$(CLUSTER_NAME) \
-		--up --down -v 10 --test=exec -- make test/e2e/testrunner-kind-$(PACKAGE)
+test/e2e/k3d-%: PACKAGE=$*
+test/e2e/k3d-%:
+	bazel build //hack/bin/... //e2e/kubetest2-k3d/...
+	PATH=bazel-bin/hack/bin:bazel-bin/e2e/kubetest2-k3d/kubetest2-k3d_/:${PATH} \
+		bazel-bin/hack/bin/kubetest2 k3d \
+		--cluster-name=$(CLUSTER_NAME) --image rancher/k3s:v1.23.3-k3s1 --servers 3 \
+		--up --down -v 10 --test=exec -- make test/e2e/testrunner-k3d-$(PACKAGE)
 
 # This target is used by kubetest2-eks to run e2e tests.
 .PHONY: test/e2e/testrunner-eks
@@ -118,7 +118,7 @@ test/e2e/testrunner-eks:
 	bazel test --stamp //e2e/decommission/...  --action_env=KUBECONFIG=$(TMPDIR)/$(CLUSTER_NAME)-eks.kubeconfig.yaml
 
 # Use this target to run e2e tests with a eks cluster.
-# This target uses kind to start a eks k8s cluster  and runs the e2e tests
+# This target uses kubetest2 to start a eks k8s cluster and runs the e2e tests
 # against that cluster.
 .PHONY: test/e2e/eks
 test/e2e/eks:
@@ -148,9 +148,9 @@ test/e2e/testrunner-gke:
 	bazel test --stamp //e2e/decommission/...
 
 # Use this target to run e2e tests with a gke cluster.
-# This target uses kind to start a gke k8s cluster  and runs the e2e tests
+# This target uses kubetest2 to start a gke k8s cluster and runs the e2e tests
 # against that cluster.
-# This is the main entrypoint for running the e2e tests on gke kind.
+# This is the main entrypoint for running the e2e tests on gke.
 # This target runs kubetest2 gke that starts a gke cluster
 # Then kubetest2 tester exec is run which runs the make target
 # test/e2e/testrunner-gke.
@@ -175,7 +175,7 @@ test/e2e/testrunner-openshift:
 	bazel test --stamp //e2e/decommission/...  --action_env=KUBECONFIG=$(HOME)/openshift-$(CLUSTER_NAME)/auth/kubeconfig
 
 # Use this target to run e2e tests with a openshift cluster.
-# This target uses kind to start a openshift cluster and runs the e2e tests
+# This target uses kubetest2 to start a openshift cluster and runs the e2e tests
 # against that cluster. A full TLD is required to creat an openshift clutser.
 # This target runs kubetest2 openshift that starts a openshift cluster
 # Then kubetest2 tester exec is run which runs the make target
@@ -257,7 +257,7 @@ dev/syncdeps:
 	@make dev/syncbazel
 
 .PHONY: dev/up
-dev/up:
+dev/up: dev/down
 	@hack/dev.sh up
 
 .PHONY: dev/down

config/templates/example.yaml.in

@@ -32,7 +32,7 @@ spec:
         volumeMode: Filesystem
   resources:
     requests:
-      # This is intentionally low to make it work on local kind clusters.
+      # This is intentionally low to make it work on local k3d clusters.
       cpu: 500m
       memory: 2Gi
     limits:

config/templates/smoketest.yaml.in

@@ -31,7 +31,7 @@ spec:
         volumeMode: Filesystem
   resources:
     requests:
-      # This is intentionally low to make it work on local kind clusters.
+      # This is intentionally low to make it work on local k3d clusters.
       cpu: 300m
       memory: 300Mi
     limits:

e2e/BUILD.bazel

@@ -21,6 +21,7 @@ filegroup(
         "//e2e/create:all-srcs",
         "//e2e/decommission:all-srcs",
         "//e2e/kubetest2-eks:all-srcs",
+        "//e2e/kubetest2-k3d:all-srcs",
         "//e2e/kubetest2-openshift:all-srcs",
         "//e2e/openshift:all-srcs",
         "//e2e/pvcresize:all-srcs",

e2e/create/create_test.go

@@ -63,7 +63,7 @@ func TestCreateInsecureCluster(t *testing.T) {
 
 	builder := testutil.NewBuilder("crdb").WithNodeCount(3).
 		WithImage(e2e.MajorVersion).WithClusterLogging("logging-configmap").
-		WithPVDataStore("1Gi", "standard" /* default storage class in KIND */)
+		WithPVDataStore("1Gi")
 
 	steps := testutil.Steps{
 		{
@@ -103,7 +103,7 @@ func TestCreatesSecureCluster(t *testing.T) {
 
 	builder := testutil.NewBuilder("crdb").WithNodeCount(3).WithTLS().
 		WithImage(e2e.MajorVersion).
-		WithPVDataStore("1Gi", "standard" /* default storage class in KIND */)
+		WithPVDataStore("1Gi")
 
 	steps := testutil.Steps{
 		{
@@ -159,7 +159,7 @@ func TestCreateSecureClusterWithInvalidVersion(t *testing.T) {
 
 					builder := testutil.NewBuilder("crdb").WithNodeCount(3).WithTLS().
 						WithImage(testcase.imageVersion).
-						WithPVDataStore("1Gi", "standard" /* default storage class in KIND */)
+						WithPVDataStore("1Gi")
 					if testcase.cockroachVersion != "" {
 						os.Setenv(relatedImageEnvName, e2e.NonExistentVersion)
 						builder = builder.WithCockroachDBVersion(testcase.cockroachVersion)
@@ -219,7 +219,7 @@ func TestCreateSecureClusterWithNonCRDBImage(t *testing.T) {
 
 					builder := testutil.NewBuilder("crdb").WithNodeCount(3).WithTLS().
 						WithImage(testcase.imageVersion).
-						WithPVDataStore("1Gi", "standard" /* default storage class in KIND */)
+						WithPVDataStore("1Gi")
 
 					if testcase.cockroachVersion != "" {
 						os.Setenv(relatedImageEnvName, e2e.InvalidImage)
@@ -265,7 +265,7 @@ func TestCreateSecureClusterWithCRDBVersionSet(t *testing.T) {
 				sb.StartManager(subT, controller.InitClusterReconcilerWithLogger(testLog))
 
 				builder := testutil.NewBuilder("crdb").WithNodeCount(3).WithTLS().
-					WithPVDataStore("1Gi", "standard" /* default storage class in KIND */).
+					WithPVDataStore("1Gi").
 					WithCockroachDBVersion(crdbVersion).WithImageObject(nil)
 
 				require.NoError(subT, sb.Create(builder.Cr()))

e2e/decommission/decommission_test.go

@@ -62,7 +62,7 @@ func TestDecommissionFunctionalityWithPrune(t *testing.T) {
 		WithNodeCount(4).
 		WithTLS().
 		WithImage(e2e.MajorVersion).
-		WithPVDataStore("1Gi", "standard" /* default storage class in KIND */)
+		WithPVDataStore("1Gi")
 
 	testutil.Steps{
 		{
@@ -124,7 +124,7 @@ func TestDecommissionFunctionality(t *testing.T) {
 		WithNodeCount(4).
 		WithTLS().
 		WithImage(e2e.MajorVersion).
-		WithPVDataStore("1Gi", "standard" /* default storage class in KIND */)
+		WithPVDataStore("1Gi")
 
 	testutil.Steps{
 		{

e2e/kubetest2-k3d/BUILD.bazel

@@ -0,0 +1,35 @@
+load("@io_bazel_rules_go//go:def.bzl", "go_binary", "go_library")
+
+go_library(
+    name = "go_default_library",
+    srcs = ["main.go"],
+    importpath = "github.com/cockroachdb/cockroach-operator/e2e/kubetest2-k3d",
+    visibility = ["//visibility:private"],
+    deps = [
+        "//e2e/kubetest2-k3d/deployer:go_default_library",
+        "@io_k8s_sigs_kubetest2//pkg/app:go_default_library",
+    ],
+)
+
+go_binary(
+    name = "kubetest2-k3d",
+    embed = [":go_default_library"],
+    visibility = ["//visibility:public"],
+)
+
+filegroup(
+    name = "package-srcs",
+    srcs = glob(["**"]),
+    tags = ["automanaged"],
+    visibility = ["//visibility:private"],
+)
+
+filegroup(
+    name = "all-srcs",
+    srcs = [
+        ":package-srcs",
+        "//e2e/kubetest2-k3d/deployer:all-srcs",
+    ],
+    tags = ["automanaged"],
+    visibility = ["//visibility:public"],
+)

e2e/kubetest2-k3d/README.md

@@ -0,0 +1,4 @@
+# kubetest2-k3d
+
+This package providers a deployer interface for kubetest2 that uses k3d. It is largely based on the kubetest2-kind
+package, but updated to use k3d for the underlying cluster.

e2e/kubetest2-k3d/deployer/BUILD.bazel

@@ -0,0 +1,37 @@
+load("@io_bazel_rules_go//go:def.bzl", "go_library")
+
+go_library(
+    name = "go_default_library",
+    srcs = [
+        "build.go",
+        "deployer.go",
+        "down.go",
+        "dumplogs.go",
+        "up.go",
+    ],
+    importpath = "github.com/cockroachdb/cockroach-operator/e2e/kubetest2-k3d/deployer",
+    visibility = ["//visibility:public"],
+    deps = [
+        "@com_github_octago_sflags//gen/gpflag:go_default_library",
+        "@com_github_spf13_pflag//:go_default_library",
+        "@io_k8s_klog//:go_default_library",
+        "@io_k8s_sigs_kubetest2//pkg/exec:go_default_library",
+        "@io_k8s_sigs_kubetest2//pkg/metadata:go_default_library",
+        "@io_k8s_sigs_kubetest2//pkg/process:go_default_library",
+        "@io_k8s_sigs_kubetest2//pkg/types:go_default_library",
+    ],
+)
+
+filegroup(
+    name = "package-srcs",
+    srcs = glob(["**"]),
+    tags = ["automanaged"],
+    visibility = ["//visibility:private"],
+)
+
+filegroup(
+    name = "all-srcs",
+    srcs = [":package-srcs"],
+    tags = ["automanaged"],
+    visibility = ["//visibility:public"],
+)