Add acceptance testing framework (#225)

* Add acceptance testing framework

* Fix compatability test conversion due to updated helm chart

* Address PR feedback and fix some cleanup issues

go.work

@@ -1,6 +1,7 @@
 go 1.22.4
 
 use (
+	./harpoon
 	./licenseupdater
 	./src/go/k8s
 )

harpoon/README.md

@@ -0,0 +1,83 @@
+# Harpoon
+
+Tools to kill a `kuttl`(fish).
+
+## Description
+
+Harpoon is a set of BDD lifecycle and Kubernetes helper wrappers around [github.com/cucumber/godog](https://github.com/cucumber/godog).
+
+## Extensions
+
+### Lifecycle Hooks
+
+Stock `godog` explicitly [removed](https://github.com/cucumber/godog/issues/335) support for any sort of feature-based hooking of their test suite steps. This makes it very difficult to have a "medium" level of isolation where some expensive to provision component is shared between features, but isolated from the rest of the suite.
+
+Currently the only way of adding in feature-based hooking is to hook into a `formatters` interface implementation that gets called when a new feature is started and reference counting the number of scenarios that will be run, and then detecting when all of those features finish.
+
+### Testing Cleanup
+
+The way that `godog` passes access to `testing.T` is weird. It passes an optional `context.Context` to every test step and you can pull a `testing.T`-like interface from the context using `godog.T(ctx)`. Unfortunately this interface doesn't have any sort of "lifecyle" helpers. Namely it doesn't expose a `Cleanup` method. This library wraps `godog.TestingT` with a cleanup manager that allows you to call teardown code on a per-scenario or per-feature basis.
+
+### Tags
+
+This library (ab)uses Gherkin tags for a couple of purposes:
+
+1. Test selection based on the concept of providers. This allows you to specify which tests run/do not run on a per-environment basis. If some features are not yet supported, or at least not yet tested on say an EKS environment, you can skip those tests when specifying a test run via `-provider eks`. This is accomplished by looking for tags in the format of `@skip:provider-name`.
+2. Shortcut test setup/teardown. This is, in part, tied to the shortcoming of `godog` in providing a mechanism for feature-level hooking. If you want to specify a feature-level before hook in `godog` you must implement it as a `Background` clause that has a singleton guard using something like `sync.Once` due to the fact that `Background` is implemented by just copying an additional step into the `Scenario` block. Rather than implementing a singleton, you can add a "tag" processor that allows you to do some sort of setup/teardown logic based on a tag (i.e. apply and delete some manifest with a given name via implementing a processer to handle `manifest:name`). Additionally, these tag processors allow you to inject additional state into context that can be fetched per test.
+
+### Kubernetes Helpers
+
+Because this is already substantially extending the footprint of `TestingT` and wrapping it in a new interface that adds lifecycle support, this interface also adds some useful helpers attached to the `TestingT` interface, namely:
+
+1. The ability to install a helm chart
+2. The ability to apply and delete manifests
+3. Embedding an initialized controller-runtime client
+
+Additionally, because the full suite setup of any Kubernetes operator generally has some typical steps, i.e. setting up dependencies. There are some top-level hooks exposed as a suite builder pattern that allows you to do things such as install CRDs or helm charts during suite initialization.
+
+### Feature Provider Table Generation
+
+Because of the tagging extensions above that allow you to skip tests on a per-provider basis, the `tablegenerator` package in this library implements a simple markdown-based table generator for documentation to show which features are supported by what providers. Its rendered output looks like this:
+
+|      SCENARIO      | EKS | AKS | GKE | K3D |
+|--------------------|-----|-----|-----|-----|
+| Some scenario      | ✅  | ✅  | ✅  | ✅  |
+
+### Normalized flag utilization
+
+Check `suite.go` for all flags, but some highlights:
+
+- `-retain`: Skip cleanup phase for tests which failed
+- `-timeout`: Set the timeout for the entire test suite to run
+- `-provider`: Specify a provider with its own unique hooks and scenarios to run
+
+## Usage
+
+Included is a "stub" test suite example, but here's a slightly more complex suite builder setup:
+
+```go
+	suite, err = framework.SuiteBuilderFromFlags().
+		RegisterProvider("eks", framework.NoopProvider).
+		RegisterProvider("gke", framework.NoopProvider).
+		RegisterProvider("aks", framework.NoopProvider).
+		RegisterProvider("k3d", framework.NoopProvider).
+		WithDefaultProvider("k3d").
+		WithHelmChart("http://some.host.local", "some", "dependency", helm.InstallOptions{
+			Name:            "dependency",
+			Namespace:       "dependency",
+			Version:         "v1.1.1",
+			CreateNamespace: true,
+		}).
+		WithCRDDirectory("path/to/crds").
+		OnFeature(func(ctx context.Context, t framework.TestingT) {
+			t.Log("Installing some helm chart")
+			t.InstallHelmChart(ctx, "http://some.host.local", "some", "chart", helm.InstallOptions{
+				Name:      "chart",
+				Namespace: t.IsolateNamespace(ctx),
+				Values: map[string]any{},
+			})
+			t.Log("Successfully installed some helm chart")
+		}).
+		RegisterTag("mytag", 1, MyTagProcessor).
+		Build()
+```

harpoon/features/stub.feature

@@ -0,0 +1,22 @@
+@isolated
+Feature: stub
+  Scenario Outline: templated stub
+    Given there is a stub
+    When a user updates the stub key "<key>" to "<value>"
+    Then the stub should have "<key>" equal "<value>"
+
+    Examples:
+        | key       | value |
+        | stub      | true  |
+
+  @isolated
+  Scenario: isolated stub
+    And there is a stub
+    When a user updates the stub key "foo" to "bar"
+    Then the stub should have "foo" equal "bar"
+
+  @skip:stub
+  Scenario: skipped stub
+    And there is a stub
+    When a user updates the stub key "a" to "b"
+    Then the stub should have "c" equal "d"

harpoon/fixtures/stub/configmap.yaml

@@ -0,0 +1,5 @@
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: stub-config-map

harpoon/go.mod

@@ -0,0 +1,178 @@
+module github.com/redpanda-data/redpanda-operator/harpoon
+
+go 1.22.4
+
+require (
+	github.com/cucumber/godog v0.14.1
+	github.com/cucumber/messages/go/v21 v21.0.1
+	github.com/olekukonko/tablewriter v0.0.5
+	github.com/redpanda-data/helm-charts v0.0.0-20240911060052-2bf9dd6f0996
+	github.com/stretchr/testify v1.9.0
+	golang.org/x/exp v0.0.0-20240904232852-e7e105dedf7e
+	k8s.io/api v0.29.5
+	k8s.io/apimachinery v0.29.5
+	k8s.io/client-go v0.29.5
+	sigs.k8s.io/controller-runtime v0.17.2
+)
+
+require (
+	github.com/AdaLogics/go-fuzz-headers v0.0.0-20230811130428-ced1acdcaa24 // indirect
+	github.com/Azure/go-ansiterm v0.0.0-20230124172434-306776ec8161 // indirect
+	github.com/BurntSushi/toml v1.3.2 // indirect
+	github.com/MakeNowJust/heredoc v1.0.0 // indirect
+	github.com/Masterminds/goutils v1.1.1 // indirect
+	github.com/Masterminds/semver/v3 v3.2.1 // indirect
+	github.com/Masterminds/sprig/v3 v3.2.3 // indirect
+	github.com/Masterminds/squirrel v1.5.4 // indirect
+	github.com/Microsoft/hcsshim v0.12.3 // indirect
+	github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 // indirect
+	github.com/beorn7/perks v1.0.1 // indirect
+	github.com/cenkalti/backoff/v4 v4.3.0 // indirect
+	github.com/cespare/xxhash/v2 v2.3.0 // indirect
+	github.com/chai2010/gettext-go v1.0.2 // indirect
+	github.com/cockroachdb/errors v1.11.1 // indirect
+	github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b // indirect
+	github.com/cockroachdb/redact v1.1.5 // indirect
+	github.com/containerd/containerd v1.7.18 // indirect
+	github.com/containerd/errdefs v0.1.0 // indirect
+	github.com/containerd/log v0.1.0 // indirect
+	github.com/creack/pty v1.1.20 // indirect
+	github.com/cucumber/gherkin/go/v26 v26.2.0 // indirect
+	github.com/cyphar/filepath-securejoin v0.2.4 // indirect
+	github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
+	github.com/distribution/distribution/v3 v3.0.0-20230821124843-59dd684cc897 // indirect
+	github.com/distribution/reference v0.6.0 // indirect
+	github.com/docker/cli v25.0.1+incompatible // indirect
+	github.com/docker/distribution v2.8.3+incompatible // indirect
+	github.com/docker/docker v27.0.3+incompatible // indirect
+	github.com/docker/docker-credential-helpers v0.8.0 // indirect
+	github.com/docker/go-connections v0.5.0 // indirect
+	github.com/docker/go-metrics v0.0.1 // indirect
+	github.com/emicklei/go-restful/v3 v3.12.0 // indirect
+	github.com/evanphx/json-patch v5.7.0+incompatible // indirect
+	github.com/evanphx/json-patch/v5 v5.9.0 // indirect
+	github.com/exponent-io/jsonpath v0.0.0-20210407135951-1de76d718b3f // indirect
+	github.com/fatih/color v1.16.0 // indirect
+	github.com/felixge/httpsnoop v1.0.4 // indirect
+	github.com/getsentry/sentry-go v0.18.0 // indirect
+	github.com/go-errors/errors v1.5.1 // indirect
+	github.com/go-gorp/gorp/v3 v3.1.0 // indirect
+	github.com/go-logr/logr v1.4.2 // indirect
+	github.com/go-logr/stdr v1.2.2 // indirect
+	github.com/go-openapi/jsonpointer v0.21.0 // indirect
+	github.com/go-openapi/jsonreference v0.21.0 // indirect
+	github.com/go-openapi/swag v0.23.0 // indirect
+	github.com/go-sql-driver/mysql v1.7.1 // indirect
+	github.com/gobwas/glob v0.2.3 // indirect
+	github.com/gofrs/uuid v4.4.0+incompatible // indirect
+	github.com/gogo/protobuf v1.3.2 // indirect
+	github.com/golang/protobuf v1.5.4 // indirect
+	github.com/gomodule/redigo v1.8.9 // indirect
+	github.com/google/btree v1.1.2 // indirect
+	github.com/google/gnostic-models v0.6.9-0.20230804172637-c7be7c783f49 // indirect
+	github.com/google/go-cmp v0.6.0 // indirect
+	github.com/google/gofuzz v1.2.0 // indirect
+	github.com/google/pprof v0.0.0-20240525223248-4bfdf5a9a2af // indirect
+	github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 // indirect
+	github.com/google/uuid v1.6.0 // indirect
+	github.com/gorilla/mux v1.8.1 // indirect
+	github.com/gorilla/websocket v1.5.0 // indirect
+	github.com/gosuri/uitable v0.0.4 // indirect
+	github.com/gregjones/httpcache v0.0.0-20190611155906-901d90724c79 // indirect
+	github.com/hashicorp/errwrap v1.1.0 // indirect
+	github.com/hashicorp/go-immutable-radix v1.3.1 // indirect
+	github.com/hashicorp/go-memdb v1.3.4 // indirect
+	github.com/hashicorp/go-multierror v1.1.1 // indirect
+	github.com/hashicorp/go-uuid v1.0.3 // indirect
+	github.com/hashicorp/golang-lru v1.0.2 // indirect
+	github.com/huandu/xstrings v1.4.0 // indirect
+	github.com/imdario/mergo v0.3.16 // indirect
+	github.com/inconshreveable/mousetrap v1.1.0 // indirect
+	github.com/jmoiron/sqlx v1.3.5 // indirect
+	github.com/josharian/intern v1.0.0 // indirect
+	github.com/json-iterator/go v1.1.12 // indirect
+	github.com/klauspost/compress v1.17.9 // indirect
+	github.com/kr/pretty v0.3.1 // indirect
+	github.com/kr/text v0.2.0 // indirect
+	github.com/lann/builder v0.0.0-20180802200727-47ae307949d0 // indirect
+	github.com/lann/ps v0.0.0-20150810152359-62de8c46ede0 // indirect
+	github.com/lib/pq v1.10.9 // indirect
+	github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de // indirect
+	github.com/mailru/easyjson v0.7.7 // indirect
+	github.com/mattn/go-colorable v0.1.13 // indirect
+	github.com/mattn/go-isatty v0.0.20 // indirect
+	github.com/mattn/go-runewidth v0.0.15 // indirect
+	github.com/mattn/go-sqlite3 v1.14.17 // indirect
+	github.com/miekg/dns v1.1.57 // indirect
+	github.com/mitchellh/copystructure v1.2.0 // indirect
+	github.com/mitchellh/go-wordwrap v1.0.1 // indirect
+	github.com/mitchellh/reflectwalk v1.0.2 // indirect
+	github.com/moby/locker v1.0.1 // indirect
+	github.com/moby/spdystream v0.4.0 // indirect
+	github.com/moby/term v0.5.0 // indirect
+	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
+	github.com/modern-go/reflect2 v1.0.2 // indirect
+	github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00 // indirect
+	github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
+	github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f // indirect
+	github.com/onsi/ginkgo/v2 v2.19.0 // indirect
+	github.com/onsi/gomega v1.34.1 // indirect
+	github.com/opencontainers/go-digest v1.0.1-0.20231025023718-d50d2fec9c98 // indirect
+	github.com/opencontainers/image-spec v1.1.0 // indirect
+	github.com/peterbourgon/diskv v2.0.1+incompatible // indirect
+	github.com/pkg/errors v0.9.1 // indirect
+	github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
+	github.com/prometheus/client_golang v1.20.0 // indirect
+	github.com/prometheus/client_model v0.6.1 // indirect
+	github.com/prometheus/common v0.55.0 // indirect
+	github.com/prometheus/procfs v0.15.1 // indirect
+	github.com/rivo/uniseg v0.4.4 // indirect
+	github.com/rogpeppe/go-internal v1.12.0 // indirect
+	github.com/rubenv/sql-migrate v1.5.2 // indirect
+	github.com/russross/blackfriday/v2 v2.1.0 // indirect
+	github.com/shopspring/decimal v1.3.1 // indirect
+	github.com/sirupsen/logrus v1.9.3 // indirect
+	github.com/spf13/cast v1.6.0 // indirect
+	github.com/spf13/cobra v1.8.1 // indirect
+	github.com/spf13/pflag v1.0.5 // indirect
+	github.com/xeipuuv/gojsonpointer v0.0.0-20190905194746-02993c407bfb // indirect
+	github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415 // indirect
+	github.com/xeipuuv/gojsonschema v1.2.0 // indirect
+	github.com/xlab/treeprint v1.2.0 // indirect
+	go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.53.0 // indirect
+	go.opentelemetry.io/otel v1.28.0 // indirect
+	go.opentelemetry.io/otel/metric v1.28.0 // indirect
+	go.opentelemetry.io/otel/trace v1.28.0 // indirect
+	go.starlark.net v0.0.0-20231121155337-90ade8b19d09 // indirect
+	golang.org/x/crypto v0.26.0 // indirect
+	golang.org/x/net v0.28.0 // indirect
+	golang.org/x/oauth2 v0.22.0 // indirect
+	golang.org/x/sync v0.8.0 // indirect
+	golang.org/x/sys v0.24.0 // indirect
+	golang.org/x/term v0.23.0 // indirect
+	golang.org/x/text v0.17.0 // indirect
+	golang.org/x/time v0.6.0 // indirect
+	google.golang.org/genproto/googleapis/rpc v0.0.0-20240808171019-573a1156607a // indirect
+	google.golang.org/grpc v1.65.0 // indirect
+	google.golang.org/protobuf v1.34.2 // indirect
+	gopkg.in/evanphx/json-patch.v5 v5.7.0 // indirect
+	gopkg.in/inf.v0 v0.9.1 // indirect
+	gopkg.in/yaml.v2 v2.4.0 // indirect
+	gopkg.in/yaml.v3 v3.0.1 // indirect
+	gotest.tools/v3 v3.5.1 // indirect
+	helm.sh/helm/v3 v3.14.4 // indirect
+	k8s.io/apiextensions-apiserver v0.29.5 // indirect
+	k8s.io/apiserver v0.29.5 // indirect
+	k8s.io/cli-runtime v0.29.5 // indirect
+	k8s.io/component-base v0.29.5 // indirect
+	k8s.io/klog/v2 v2.130.1 // indirect
+	k8s.io/kube-openapi v0.0.0-20240411171206-dc4e619f62f3 // indirect
+	k8s.io/kubectl v0.29.5 // indirect
+	k8s.io/utils v0.0.0-20240711033017-18e509b52bc8 // indirect
+	oras.land/oras-go v1.2.5 // indirect
+	sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd // indirect
+	sigs.k8s.io/kustomize/api v0.16.0 // indirect
+	sigs.k8s.io/kustomize/kyaml v0.17.2 // indirect
+	sigs.k8s.io/structured-merge-diff/v4 v4.4.1 // indirect
+	sigs.k8s.io/yaml v1.4.0 // indirect
+)