diff --git a/README.md b/README.md index aabd3878..9c446ab5 100644 --- a/README.md +++ b/README.md @@ -13,7 +13,7 @@ KUTTL is designed for testing operators, however it can declaratively test any k ## Getting Started -Please refer to the [getting started guide](https://kuttl.dev/docs/) documentation. +Please refer to the [getting started guide](docs/README.md) documentation. ## Resources @@ -36,7 +36,7 @@ Learn more on how to engage with the KUDO community on the [community page](http ## Contributions -Please read the [contributing guide](https://github.com/kudobuilder/kuttl/blob/main/CONTRIBUTING.md) for details around: +Please read the [contributing guide](CONTRIBUTING.md) for details around: 1. Code of Conduct 1. Code Culture diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 00000000..6518aacb --- /dev/null +++ b/docs/README.md @@ -0,0 +1,24 @@ +# Getting Started + +## Pre-requisites + +Before you get started using KUTTL, you need to have a running Kubernetes cluster setup. If you already have a cluster there are no prerequisites. If you want to use the mocked control plane or Kind, you will need [Kind](https://github.com/kubernetes-sigs/kind). + +- Setup a Kubernetes Cluster in version `1.13` or later +- Install [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) in version `1.13` or later. + +## Install KUTTL CLI + +Install the `kubectl kuttl` plugin. To do so, please follow the [CLI plugin installation instructions](cli.md). + +The KUTTL CLI leverages the kubectl plugin system, which gives you all its functionality under `kubectl kuttl`. + +## Using KUTTL + +Once you have a running cluster with `kubectl` installed along with the KUTTL CLI plugin, you can run tests with KUTTL like so: + +```bash +$ kubectl kuttl test path/to/test-suite +``` + +[Learn more](what-is-kuttl.md) about KUTTL and check out how to get started with the [KUTTL test harness](kuttl-test-harness.md). diff --git a/docs/api-integration.md b/docs/api-integration.md new file mode 100644 index 00000000..90655530 --- /dev/null +++ b/docs/api-integration.md @@ -0,0 +1,53 @@ +# API Integration + +It is possible to integrate KUTTL into your own Go test infrastructure. KUDO provides as an example `kubectl kudo test` using the KUTTL test harness. The following are the necessary steps. + +## Add KUTTL to Go.mod + +`go get github.com/kudobuilder/kuttl` + +or get a specific version + +`go get github.com/kudobuilder/kuttl@v0.1.0` + +## Common Imports to Use + +The test harness type is defined in an `apis` package similar to a Kubernetes type along with a version package. The test harness is currently `v1beta1` and provides the main configuration for a test suite. + +The `test` package contains the `test.Harness` implementation (given the configuration of the test harness configuration type previously mentioned). The `test.Harness` provides the "run" of the test run and needs a Go `t *testing.T`. + +The `testutils` package contains utilities for docker, kubernetes, loggers and testing. + +```go +import ( + harness "github.com/kudobuilder/kuttl/pkg/apis/testharness/v1beta1" + "github.com/kudobuilder/kuttl/pkg/test" + testutils "github.com/kudobuilder/kuttl/pkg/test/utils" +) +``` + +## Test Harness + +The `harness.TestSuite` is the structure that controls how the test harness will run. + +```go +options := harness.TestSuite{} +``` + +The Go `t *testing.T` and `harness.TestSuite` are provided to `test.Harness` which provides the implementation for testing. + +```go +Run: func(cmd *cobra.Command, args []string) { + testutils.RunTests("kudo", testToRun, options.Parallel, func(t *testing.T) { + harness := test.Harness{ + TestSuite: options, + T: t, + } + + harness.Run() + }) +}, + +``` + +A more complete example is provided in KUDOs [cmd/test.go](https://github.com/kudobuilder/kudo/blob/master/pkg/kudoctl/cmd/test.go) diff --git a/docs/cli.md b/docs/cli.md new file mode 100644 index 00000000..4b679484 --- /dev/null +++ b/docs/cli.md @@ -0,0 +1,151 @@ +# CLI Usage + +This document demonstrates how to use the KUTTL CLI + +## Setup the KUTTL Kubectl Plugin + +### Requirements + +- `kubectl` version `1.13.0` or newer + +### Installation + +You can either download CLI binaries for linux or MacOS from our [release page](https://github.com/kudobuilder/kuttl/releases), or install the CLI plugin using `brew`: + +```bash +brew tap kudobuilder/tap +brew install kuttl-cli +``` + +or you can compile and install the plugin from your `$GOPATH/src/github.com/kudobuilder/kuttl` root folder via: + +```bash +make cli-install +``` + +Another alternative is [`krew`](https://github.com/kubernetes-sigs/krew), the package manager for kubectl plugins. + +```bash +kubectl krew install kuttl +``` + +## Commands + +* **`kubectl kuttl help [command] [flags]`** + + Provide general help or help on a specific command + +* **`kubectl kuttl version`** + + Print the current KUTTL version. + +* **`kubectl kuttl test`** + + Run KUTTL test harness. + + +## Flags + +> [!NOTE] +> **Usage** +> +> `kubectl kuttl test [flags]` + +Flags are: + +* **`-h, --help`** + + Help for test + +* **`--artifacts-dir (string)`** + + Directory to output kind logs to (if not specified, the current working directory). + +* **`--config (string)`** + + Path to file to load test settings from. This is usually the `kuttl-test.yaml` file. + +* **`--crd-dir (string)`** + + Directory to load CustomResourceDefinitions from prior to running the tests. + +* **`--kind-config (string)`** + + Specify the KIND configuration file path (implies `--start-kind`, cannot be used with `--start-control-plane`). + +* **`--kind-context (string)`** + + Specify the KIND context name to use (default: `kind`). + +* **`--manifest-dir (stringArray)`** + + One or more directories containing manifests to apply before running the tests. + +* **`--parallel (int)`** + + The maximum number of tests to run at once. (default `8`) + +* **`--skip-cluster-delete (bool)`** + + If set, do not delete the mocked control plane or kind cluster. + +* **`--skip-delete (bool)`** + + If set, do not delete resources created during tests (helpful for debugging test failures, implies `--skip-cluster-delete`). + +* **`--start-control-plane (bool)`** + + Start a local Kubernetes control plane for the tests (requires `etcd` and `kube-apiserver` binaries, cannot be used with `--start-kind`). + +* **`--start-kind (bool)`** + + Start a KIND cluster for the tests (cannot be used with `--start-control-plane`). + +* **`--test (string)`** + + If set, the specific test case to run. + +* **`--test-run-labels (string)`** + + Optional label set to associate with this test run. + This label set can then be matched against by the `testRunSelector` in `TestFile` objects to optionally exclude selected files. + The syntax is comma-separated list of `key=value` assignments. + +* **`-v or -vv (int)`** + Logging verbosity level. 0=normal, 1=verbose, 2=detailed, 3 or more =trace. + + + +## Examples + +### KUTTL Test + +KUTTL test command is the heart of the test harness. It requires a `kuttl-test.yaml` which defines the test setup. + +```yaml +apiVersion: kuttl.dev/v1beta1 +kind: TestSuite +testDirs: +- ./test/integration +parallel: 4 +``` + +The default can be run as follows: + +```bash +kubectl kuttl test pkg/test/test_data/ +``` + +When running with no defined [test environment](testing/test-environments.md), the default is a preconfigured cluster defined in `$KUBECONFIG`. + +To run with the mocked control plane run: + +```bash +kubectl kuttl test --start-control-plane pkg/test/test_data/ +``` + +In order to run with the full kind cluster stack, run: + +```bash +kubectl kuttl test --start-kind pkg/test/test_data/ +``` diff --git a/docs/contributing.md b/docs/contributing.md new file mode 100644 index 00000000..757e97f1 --- /dev/null +++ b/docs/contributing.md @@ -0,0 +1,38 @@ +--- +title: Contributing +type: docs +menu: docs +--- + +# Contributing + +The source code for [KUTTL](https://github.com/kudobuilder/kuttl) lives on GitHub. We welcome feature requests and bug reports in the form of [issues](https://help.github.com/en/articles/about-issues), and of course code - which includes documentation! - in the form of [pull requests](https://help.github.com/en/articles/about-pull-requests) (PRs). + +There's a ton of stuff to do and there's opportunities to contribute in a variety of ways. We'd suggest that newcomers look at issues tagged with ['good first issue'](https://github.com/kudobuilder/kuttl/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22) and ['help wanted'](https://github.com/kudobuilder/kuttl/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22) and then then jump into [#kudo on the Kubernetes Slack](https://kubernetes.slack.com/messages/kudo/) to discuss (join Kubernetes slack via [self-invitation link](https://slack.k8s.io/) if you don't have an account yet). KUTTL is now separate from KUDO, however we currently plan to use the same communication channels. + +Please also take some time to read our [Contributing Guidelines](https://github.com/kudobuilder/kuttl/blob/master/CONTRIBUTING.md). + +## Raising an Issue + +If you've hit a bug, have an idea for a new feature, or want to suggest some other kind of change then we welcome an issue detailing your problem or your suggestion. Ideally we'd ask that people reach out in the Slack channel and join one of our weekly meetings so that other developers and users can help iterate. + +## Creating a Pull Request + +Yes please! Bring us your code! There's a whole lot of work to do, and we're committed to building an active community around KUTTL in order to ensure its longevity. + +PRs raised against either repo have a default template which help guide contributors to focus on the details necessary for a speedy review. Again, please follow-up with discussion in the Slack channel. We're also happy for people to submit draft PRs which can then be worked through with other members of the KUDO/KUTTL community. + +## Reviewing a Pull Request + +This process is adapted from the one defined for [contributing to Kubernetes](https://kubernetes.io/docs/contribute/intermediate/#review-a-pr) itself, so should be familiar. + +* Examine the PR description and read any associated issues or links for context; +* Look over all changed files, and if you have a comment or a question on any highlighted section then start a review; +* Continue to add comments using this review process and when you've finished, choose either 'comment' for general commentary or 'request changes' for anything you deem important enough to warrant further work; +* If you spot a relatively trivial error such as a typo or something that's not directly related to the stated purpose of the PR then you can let the submitter know by prefixing your review comment with `nit:`. These are not necessarily blockers to the PR itself but it gives the author an opportunity to make amendments; +* If you think the PR is ready to be merged, then you can add the command `/approve` to your summary comment. Note that only those listed in the approvers section of the [OWNERS](https://github.com/kudobuilder/kudo/blob/master/OWNERS) file can use this command; +* PRs can be assigned to an individual with the `/assign` command. If you think a proposed change needs a specific person's input, use this command along with their GitHub username to get their attention; +* If a PR has the `lgtm` and / or the `approve` label then it will be merged automatically; + * You can apply the `do-not-merge/hold` label in order to stop PRs from being merged automatically. + +Typically, a PR needs a review and an approval from two [core developers](https://github.com/orgs/kudobuilder/people) in order to be merged. diff --git a/docs/kuttl-test-harness.md b/docs/kuttl-test-harness.md new file mode 100644 index 00000000..9e3508b2 --- /dev/null +++ b/docs/kuttl-test-harness.md @@ -0,0 +1,160 @@ +# KUTTL Test Harness + +KUTTL is a declarative integration testing harness for testing operators, KUDO, [Helm charts](testing/tips.md#helm-testing), and any other Kubernetes applications or controllers. Test cases are written as plain Kubernetes resources and can be run against a mocked control plane, locally in kind, or any other Kubernetes cluster. + +Whether you are developing an application, controller, operator, or deploying Kubernetes clusters the KUTTL test harness helps you easily write portable end-to-end, integration, and conformance tests for Kubernetes without needing to write any code. + +## Installation + +The test harness CLI is included in the KUTTL CLI, to install we can install the CLI using [krew](https://github.com/kubernetes-sigs/krew): + +```bash +krew install kuttl +``` + +You can now invoke the KUDO test CLI: + +```bash +kubectl kuttl test --help +``` + +See the [KUTTL installation guide](cli.md#installation) for alternative installation methods. + +## Writing Your First Test + +Now that the KUTTL CLI is installed, we can write a test. The KUTTL test CLI organizes tests into suites: + +* A "test step" defines a set of Kubernetes manifests to apply and a state to assert on (wait for or expect). +* A "test case" is a collection of test steps that are run serially - if any test step fails then the entire test case is considered failed. +* A "test suite" is comprised of many test cases that are run in parallel. +* The "test harness" is the tool that runs test suites (the KUTTL CLI). + +Be aware that KUTTL CLI expects a kuttl-test.yaml needs to be available, see [setup the kuttl kubectl plugin](cli.md#setup-the-kuttl-kubectl-plugin) if you didn't do so yet. + +### Create a Test Case + +First, let's create a directory for our test suite, let's call it `tests/e2e`: + +```sh +mkdir -p tests/e2e +``` + +Next, we'll create a directory for our test case, the test case will be called `example-test`: + +```bash +mkdir tests/e2e/example-test +``` + +Inside of `tests/e2e/example-test/` create our first test step, `00-install.yaml`, which will create a deployment called `example-deployment`: + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: example-deployment + labels: + app: nginx +spec: + replicas: 3 + selector: + matchLabels: + app: nginx + template: + metadata: + labels: + app: nginx + spec: + containers: + - name: nginx + image: nginx:latest + ports: + - containerPort: 80 +``` + +Note that in this example, the deployment does not have a `namespace` set. The test harness will create a namespace for each test case and run all of the test steps inside of it. However, if a resource already has a namespace set (or is not a namespaced resource), then the harness will respect the namespace that is set. + +Each filename in the test case directory should start with an index (in this example `00`) that indicates which test step the file is a part of. Files that do not start with a step index are ignored and can be used for documentation or other test data. Test steps are run in order and each must be successful for the test case to be considered successful. + +Now that we have a test step, we need to create a test assert. The assert's filename should be the test step index followed by `-assert.yaml`. Create `tests/e2e/example-test/00-assert.yaml`: + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: example-deployment +status: + readyReplicas: 3 +``` + +This test step will be considered completed once the pod matches the state that we have defined. If the state is not reached by the time the assert's timeout has expired (30 seconds, by default), then the test step and case will be considered failed. + +### Run the Tests + +Let's run this test suite: + +```sh +kubectl kuttl test --start-kind=true ./tests/e2e/ +``` + +Running this command will: + +* Start a [kind (Kubernetes-in-Docker) cluster](https://github.com/kubernetes-sigs/kind), if there is not already one running. +* Create a new namespace for the test case. +* Create the resources defined in `tests/e2e/example-test/00-install.yaml`. +* Wait for the state defined in `tests/e2e/example-test/00-assert.yaml` to be reached. +* Collect the kind cluster's logs. +* Tear down the kind cluster (or you can run `kubectl kuttl test` with `--skip-cluster-delete` to keep the cluster around after the tests run). + +### Write a Second Test Step + +Now that we have successfully written a test case, let's add another step to it. In this step, let's increase the number of replicas on the deployment we created in the first step from 3 to 4. + +Create `tests/e2e/example-test/01-scale.yaml`: + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: example-deployment +spec: + replicas: 4 +``` + +Now create an assert for it in `tests/e2e/example-test/01-assert.yaml`: + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: example-deployment +status: + readyReplicas: 4 +``` + +Run the test suite again and the test will pass: + +```sh +kubectl kuttl test --start-kind=true ./tests/e2e/ +``` + +### Test Suite Configuration + +To add this test suite to your project, create a `kuttl-test.yaml` file: + +```yaml +apiVersion: kuttl.dev/v1beta1 +kind: TestSuite +testDirs: +- ./tests/e2e/ +startKIND: true +``` + +Now we can run the tests just by running `kubectl kuttl test` with no arguments. + +Any arguments provided on the command line will override the settings in the `kuttl-test.yaml` file, e.g. to skip using kind and run the tests against a live Kubernetes cluster, run: + +```sh +kubectl kuttl test --start-kind=false +``` + +Now that your first test suite is configured, see [test environments](testing/test-environments.md) for documentation on customizing your test environment or the [test step documentation](testing/steps.md) to write more advanced tests. diff --git a/docs/testing/asserts-errors.md b/docs/testing/asserts-errors.md new file mode 100644 index 00000000..b22ec0e7 --- /dev/null +++ b/docs/testing/asserts-errors.md @@ -0,0 +1,120 @@ +# Asserts and Errors + +Test asserts are the part of a [test step](steps.md) that define the state to wait for Kubernetes to reach. It is possible to match specific objects by name as well as match any object that matches a defined state. Test errors define states that should not be reached. + +## Format + +The test assert file for a test step is found at `$index-assert.yaml`. So, if the test step index is `00`, the assert should be called `00-assert.yaml`. This file can contain any number of objects to match on. If the objects have a namespace set, it will be respected, but if a namespace is not set, then the test harness will look for the objects in the test case's namespace. + +The test error file for a test step is found at `$index-errors.yaml` and works similar to the test assert file. + +By default, a test step will wait for up to 30 seconds for the defined state to be reached. See the [configuration reference](reference.md#testassert) for documentation on configuring test asserts. + +Note that an assertion or errors file is optional. If absent, the test step will be considered successful immediately once the object(s) in the test step have been created. It is also valid to create a test step that does not create any objects, but only has an assertion or errors file. + +## Getting a Resource from the Cluster + +If an object has a name set, then the harness will look specifically for that object to exist and then verify that its state matches what is defined in the assert file. For example, if the assert file has: + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: my-pod +status: + phase: Successful +``` + +Then the test harness will wait for the `my-pod` pod in the test namespace to have `status.phase=Successful`. Note that any fields *not* specified in the assert file will be ignored, making it possible to specify only the important fields for the test step. + +If this object is in the errors file, the test harness will report an error if that object exists and its state matches what is defined in the errors file. + +## Listing Resources in the Cluster + +If an object in the assert file has no name set, then the harness will list objects of that kind. +If the object in the assert file has `metadata.labels` field, then it will be used as a label selector for the list operation. +Then `kuttl` will expect there to be at least one object that matches. For example, an assert: + +```yaml +apiVersion: v1 +kind: Pod +metadata: + labels: + app: my-app +status: + phase: Successful +``` + +This example would wait for a pod with an `app` label value of `my-app` to exist in the test namespace with the `status.phase=Successful`. + +If no labels were specified, *any* pod with specified status would satisfy the assertion. + +If this is defined in the errors file instead, the test harness will report an error if *any* such pod exists in the test namespace with `status.phase=Successful`. + +## Failures + +When a failure occurs in either an `assert` or `errors` step, kuttl will print a difference (diff) in the test output showing the reason why the step was deemed to fail. While this may be helpful in most cases, it may still be insufficient to determine the exact cause of a failure. Some additional information may be required to fully explain why a step failed which provides fuller context. When the diff is not adequate to explain a failure, a [`collectors`](reference.md#collectors) object may optionally be used to gather further troubleshooting information in the form of pod logs, namespace events, or output of a command. + +For example, consider a simple test case in which a pod is created as the initial step followed by an assertion that the pod is present and in a state of `ready=true`. If the pod is observed to contain the state `ready=false` the step, and test, will fail. With a `collectors` object present in the `TestAssert`, it may provide logs for the pod to help explain why this state was not reached. + +`01-pod.yaml` + +```yaml +apiVersion: v1 +kind: Pod +metadata: + labels: + run: hello-world + name: hello-world + namespace: default +spec: + containers: + - image: docker.io/hello-world + name: hello-world +``` + +`01-assert.yaml` + +```yaml +apiVersion: kuttl.dev/v1beta1 +kind: TestAssert +timeout: 5 +collectors: +- type: pod + pod: hello-world + namespace: default +--- +apiVersion: v1 +kind: Pod +metadata: + labels: + run: hello-world + name: hello-world + namespace: default +status: + running: true +``` + +In this example, the `hello-world` container was not started with any arguments resulting in its running followed by termination as expected. Therefore, the status of `running=true` was not asserted. + +In the command output, prior to the full diff kuttl displays will be shown the pod's logs. + +```log + logger.go:42: 20:06:29 | collectors/1-pod | starting test step 1-pod + logger.go:42: 20:06:30 | collectors/1-pod | Pod:default/hello-world created + logger.go:42: 20:06:35 | collectors/1-pod | test step failed 1-pod + logger.go:42: 20:06:35 | collectors/1-pod | collecting log output for [type==pod,pod==hello-world,namespace: default] + logger.go:42: 20:06:35 | collectors/1-pod | running command: [kubectl logs --prefix hello-world -n default --all-containers --tail=-1] + logger.go:42: 20:06:35 | collectors/1-pod | [pod/hello-world/hello-world] + logger.go:42: 20:06:35 | collectors/1-pod | [pod/hello-world/hello-world] Hello from Docker! + logger.go:42: 20:06:35 | collectors/1-pod | [pod/hello-world/hello-world] This message shows that your installation appears to be working correctly. + logger.go:42: 20:06:35 | collectors/1-pod | [pod/hello-world/hello-world] + logger.go:42: 20:06:35 | collectors/1-pod | [pod/hello-world/hello-world] To generate this message, Docker took the following steps: + logger.go:42: 20:06:35 | collectors/1-pod | [pod/hello-world/hello-world] 1. The Docker client contacted the Docker daemon. + logger.go:42: 20:06:35 | collectors/1-pod | [pod/hello-world/hello-world] 2. The Docker daemon pulled the "hello-world" image from the Docker Hub. + + case.go:362: failed in step 1-pod + case.go:364: --- Pod:default/hello-world +``` + +See the [reference page](reference.md#collectors) for more configuration options available with the `collectors` object. diff --git a/docs/testing/reference.md b/docs/testing/reference.md new file mode 100644 index 00000000..228103d6 --- /dev/null +++ b/docs/testing/reference.md @@ -0,0 +1,165 @@ +# KUTTL Configuration Reference + +## TestSuite + +The `TestSuite` object specifies the settings for the entire test suite and should live in the test suite configuration file (`kuttl-test.yaml` by default, or `--config`): + +```yaml +apiVersion: kuttl.dev/v1beta1 +kind: TestSuite +startKIND: true +kindContainers: +- your/image:latest +testDirs: +- tests/e2e/ +timeout: 120 +``` + +Supported settings: + +Field | Type | Description | Default +------------------|------------------|------------------------------------------------------------------------------------------|-------- +crdDir | string | Path to CRDs to install before running tests. KUTTL waits for CRDs to be available prior to starting tests. | +manifestDirs | list of strings | Paths to manifests to install before running tests. | +testDirs | list of strings | Directories containing test cases to run. | +startControlPlane | bool | Whether or not to start a local etcd and kubernetes API server for the tests. | false +startKIND | bool | Whether or not to start a local kind cluster for the tests. | false +kindNodeCache | bool | If set, each node defined in the kind configuration will have a docker volume mounted into it to persist pulled container images across test runs | false +kindConfig | string | Path to the KIND configuration file to use. | +kindContext | string | KIND context to use. | "kind" +skipDelete | bool | If set, do not delete the resources after running the tests (implies SkipClusterDelete). | false +skipClusterDelete | bool | If set, do not delete the mocked control plane or kind cluster. | false +timeout | int | Override the default timeout of 30 seconds (in seconds). | 30 +parallel | int | The maximum number of tests to run at once. | 8 +artifactsDir | string | The directory to output artifacts to (current working directory if not specified). | . +commands | list of [Commands](#commands) | Commands to run prior to running the tests. | [] +kindContainers | list of strings | List of Docker images to load into the KIND cluster once it is started. | [] +reportFormat | string | Determines the report format. If empty, no report is generated. One of: JSON, XML. | +reportName | string | The name of report to create. This field is not used unless reportFormat is set. | "kuttl-test" +namespace | string | The namespace to use for tests. This namespace will be created if it does not exist and removed if it was created (unless `skipDelete` is set). If no namespace is set, one will be auto-generated. | +suppress | list of strings | Suppresses log collection of the specified types. Currently only `events` is supported. | + +## TestStep + +The `TestStep` object can be used to specify settings for a test step and can be specified in any test step YAML. + +```yaml +apiVersion: kuttl.dev/v1beta1 +kind: TestStep +apply: +- my-new-resource.yaml +assert: +- my-asserted-new-resource.yaml +error: +- my-errored-new-resource.yaml +unitTest: false +delete: +- apiVersion: v1 + kind: Pod + name: my-pod +commands: +- command: helm init +kubeconfig: foo.kubeconfig +``` + +Supported settings: + +Field | Type | Description +---------|---------------------------|--------------------------------------------------------------------- +apply | list of files | A list of files to apply as part of this step. Specified path is relative to that in which the step occurs. +assert | list of files | A list of files to assert as part of this step. See documentation for [asserts and errors](asserts-errors.md) for more information. Specified path is relative to that in which the step occurs. +error | list of files | A list of files to error as part of this step. See documentation for [asserts and errors](asserts-errors.md) for more information. Specified path is relative to that in which the step occurs. +delete | list of object references | A list of objects to delete, if they do not already exist, at the beginning of the test step. The test harness will wait for the objects to be successfully deleted before applying the objects in the step. +index | int | Override the test step's index. +commands | list of [Commands](#commands) | Commands to run prior at the beginning of the test step. +kubeconfig | string | The Kubeconfig file to use to run the included steps(s). +unitTest | bool | Indicates if the step is a unit test, safe to run without a real Kubernetes cluster. + + +Object Reference: + +Field | Type | Description +-----------|--------|--------------------------------------------------------------------- +apiVersion | string | The Kubernetes API version of the objects to delete. +kind | string | The Kubernetes kind of the objects to delete. +name | string | If specified, the name of the object to delete. If not specified, all objects that match the specified labels will be deleted. +namespace | string | The namespace of the objects to delete. +labels | map | If specified, a label selector to use when looking up objects to delete. If both labels and name are unspecified, then all resources of the specified kind in the namespace will be deleted. + +## TestAssert + +The `TestAssert` object can be used to specify settings for a test step's assert and must be specified in the test step's assert YAML. + +```yaml +apiVersion: kuttl.dev/v1beta1 +kind: TestAssert +timeout: 30 +commands: +- command: echo hello +collectors: +- type: pod + pod: nginx +``` + +Supported settings: + +Field | Type | Description | Default +--------|------|-------------------------------------------------------|------------- +timeout | int | Number of seconds that the test is allowed to run for | 30 +collectors | list of [collectors](#collectors) | The collectors to be invoked to gather information upon step failure | N/A +commands | list of [commands](#commands) | Commands to run prior to the beginning of the test step. | N/A + +## TestFile + +A `TestFile` object can be used to provide configuration concerning a single YAML test file that contains it. + +```yaml +apiVersion: kuttl.dev/v1beta1 +kind: TestFile +testRunSelector: + matchLabels: + flavor: vanilla +``` + +Supported settings: + +| Field | Type | Description | Default | +|-----------------|----------------|---------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------| +| testRunSelector | label selector | If this selector does not match [labels of this test run](#test-run-labels-and-selectors), the containing file will be ignored. | Empty label selector (matches all possible test label sets). | + + +### Test Run Labels and Selectors + +An invocation of `kuttl test` may specify a label set associated with a test run using a command line flag. +One can then use a `TestFile` object with `testRunSelector` to decide whether a given test YAML file should be included +in a test run or not. + +## Collectors + +The `Collectors` object is used by the `TestAssert` object as a way to collect certain information about the outcome of an `assert` or `errors` step should it fail. A collector is only invoked in cases where a failure occurs and not if the step succeeds. Collection can occur from Pod logs, Namespace events, or the output of a custom command. + +Supported settings: + +Field | Type | Description | Default +--------|------|-------------------------------------------------------|------------- +type | string | Type of collector to run. Values are one of `pod`, `command`, or `events`. If the field named `command` is specified, `type` is assumed to be `command`. If the field named `pod` is specified, `type` is assumed to be `pod`. | `pod` +pod | string | The pod name from which to access logs. | N/A +namespace | string | Namespace in which the pod or events can be located. | N/A +container | string | Container name inside the pod from which to fetch logs. If empty assumes all containers. | unset +selector | string | Label query to select a pod. | N/A +tail | int | The number of last lines to collect from a pod. | 10 (if selector); all (if pod name) +command | string | Command to run. Requires an empty type or type `command`. Must not specify fields `pod`, `namespace`, `container`, or `selector` if present. | N/A + +## Commands + +The `Commands` object is used by `TestStep`, `TestAssert`, and `TestSuite` to enable running commands in tests: + +Field | Type | Description +--------------|--------|--------------------------------------------------------------------- +command | string | The command and argument to run as a string. +script | string | Allows a shell script to run - namespaced and command should not be used with script. namespaced is ignored and command is an error. env expansion is depended upon the shell but ENV is passed to the runtime env. +namespaced | bool | If set, the `--namespace` flag will be appended to the command with the namespace to use (the test namespace for a test step or "default" for the test suite). +ignoreFailure | bool | If set, failures will be ignored. +background | bool | If this command is to be started in the background. These are only support in TestSuites. +skipLogOutput | bool | If set, the output from the command is *not* logged. Useful for sensitive logs or to reduce noise. +timeout | int | Override the TestSuite timeout for this command (in seconds). diff --git a/docs/testing/steps.md b/docs/testing/steps.md new file mode 100644 index 00000000..6eea9c2c --- /dev/null +++ b/docs/testing/steps.md @@ -0,0 +1,132 @@ +# Steps + +Each test case is broken down into test steps. Test steps within a test case are run sequentially: if any of the test steps fail, the entire test case is considered failed. + +A test step can create, update, and delete objects as well as run any kubectl command. + +## Format + +A test step can include many YAML files and each YAML file can contain many Kubernetes objects. In a test case's directory, each file that begins with the same index is considered a part of the same test step. All objects inside of a test step are operated on by the test harness simultaneously, so use separate test steps to order operations. + +E.g., in a test case directory: + +```text +tests/e2e/example/00-pod.yaml +tests/e2e/example/00-example.yaml +tests/e2e/example/01-staging.yaml +``` + +There are two test steps: + +* `00`, which includes `00-pod.yaml` and `00-example.yaml`. +* `01`, which includes `01-staging.yaml`. + +The test harness would run test step `00` and once completed, run test step `01`. + +A namespace is created by the test harness for each test case, so if an object in the step does not have a namespace set, then it will be created in the test case's namespace. If a namespace is set, then that namespace will be respected throughout the tests (making it possible to test resources that reside in standardized namespaces). + +See the [configuration reference](reference.md#teststep) for documentation on configuring test steps. + +## Creating Objects + +Any objects specified in a test step will be created if they do not already exist. + +## Updating Objects + +If an object does already exist in Kubernetes, then the object in Kubernetes will be updated with the changes specified. + +The test harness uses merge patching for updating objects, so it is possible to specify minimal updates. For example, to change the replicas on a deployment but leave all other settings untouched, a step could be written: + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: my-deployment +spec: + replicas: 4 +``` + +## Deleting Objects + +To delete objects at the beginning of a test step, you can specify object references to delete in your `TestStep` configuration. In a test step file, add a `TestStep` object: + +```yaml +apiVersion: kuttl.dev/v1beta1 +kind: TestStep +delete: +# Delete a Pod +- apiVersion: v1 + kind: Pod + name: my-pod +# Delete all Pods with app=nginx +- apiVersion: v1 + kind: Pod + labels: + app: nginx +# Delete all Pods in the test namespace +- apiVersion: v1 + kind: Pod +``` + +The `delete` object references can delete: + +* A single object by specifying its `name`. +* If `labels` is set and `name` is omitted, then objects matching the labels and kind will be deleted. +* If both `name` and `labels` omitted, all objects of the specified kind in the test namespace will be deleted. + +The test harness will wait for the objects to be successfully deleted, if they exist, before continuing with the test step - if the objects do not get deleted before the timeout has expired the test step is considered failed. + +## Running Commands + +A `TestStep` configuration can also specify commands to run before running the step: + +```yaml +apiVersion: kuttl.dev/v1beta1 +kind: TestStep +commands: + - command: kubectl apply -f https://raw.githubusercontent.com/kudobuilder/kudo/master/docs/deployment/10-crds.yaml + namespaced: true +``` + +If the `namespaced` setting is set, the `--namespace` flag is set to the test step's namespace. + +It is also possible to use any installed kubectl plugin when calling kubectl commands: + +```yaml +apiVersion: kuttl.dev/v1beta1 +kind: TestStep +commands: + - command: kubectl kudo install zookeeper --skip-instance +``` + +Use defining commands, it is possible to use shell expansion in the command or the scripts the command calls. Command expansion is the replacement of a variable beginning with `$` with a value from the env such as `$HOME`. Expands include the OS environment variables. In addition KUTTL provides or replaces the following: + +- `$NAMESPACE` is the namespace kuttl is running the test under +- `$PATH` KUTTL prepends the $PATH with the `$CWD/bin` +- `$KUBECONFIG` is the `$CWD/kubeconfig` + +> [!WARNING] +> **Command Expansion of `$`** +> +> The `$` in the command signifies the need for an expansion. +> If you have a need to use `$` without expansion, you will need to escape it by expressing `$$`, +> which will result in a single `$` when the command runs. + +### Shell scripts + +The command allows only a single binary with parameters to be executed. It does not allow any shell scripting, especially pipes to be used. For bigger problems, it is fine to write a custom shell script and call this via the command, but for simple tasks, KUTTL allows the use of an inline script: + +```yaml +apiVersion: kuttl.dev/v1beta1 +kind: TestStep +commands: + - script: kubectl kudo init --upgrade --dry-run --output yaml | kubectl delete -f - +``` + +When `script` instead of `command` is used, the attributes `namespaced` is not allowed and silently ignored. You can however use the `$NAMESPACE` environment variable as well as all other ENV vars. + +> [!WARNING] +> **Shell dependent behavior** +> +> Scripts are executed by prepending `sh -c` to the given script +> and therefore their behavior depends on the configured environment and shell. diff --git a/docs/testing/test-environments.md b/docs/testing/test-environments.md new file mode 100644 index 00000000..47db3ad3 --- /dev/null +++ b/docs/testing/test-environments.md @@ -0,0 +1,152 @@ +# Test Environments + +The KUTTL test harness can run tests against several different test environments, allowing your test suites to be used in many different environments. + +A default environment for the tests can be defined in `kuttl-test.yaml` allowing each test suite or project to easily use the correct environment. + +## Live Cluster + +If no configuration is provided, the tests will run against your default cluster context using whatever Kubernetes cluster is configured in your kubeconfig. + +You can also provide an alternative kubeconfig file by either setting `$KUBECONFIG` or the `--kubeconfig` flag: + +```bash +kubectl kuttl test --kubeconfig=mycluster.yaml +``` + +## Kubernetes-in-docker + +KUTTL has a built in integration with [kind](https://github.com/kubernetes-sigs/kind) to start and interact with kubernetes-in-docker clusters. + +To start a kind cluster in your tests either specify it on the command line: + +```bash +kubectl kuttl test --start-kind=true +``` + +Or specify it in your `kuttl-test.yaml`: + +```yaml +apiVersion: kudo.k8s.io/v1alpha1 +kind: TestSuite +kindNodeCache: true +``` + +By default KUTTL will use the default kind cluster name of "kind". If a kind cluster is already running with that name, it will use the existing cluster. + +The kind cluster name can be overridden by setting either `kindContext` in your configuration or `--kind-context` on the command line. + +By setting `kindNodeCache`, the containerd directories will be mounted into a Docker volume in order to persist the images pulled during a test run across test runs. + +If you want to load images into the built KIND cluster that have not been pushed, set `kindContainers`. See [Tips And Tricks](tips.md#loading-built-images-into-kind) for an example. + +It is also possible to provide a custom kind configuration file. For example, to override the Kubernetes cluster version, create a kind configuration file called `kind.yaml`: + +```yaml +kind: Cluster +apiVersion: kind.sigs.k8s.io/v1alpha3 +nodes: +- role: control-plane + image: kindest/node:v1.14.3 +``` + +See the [kind documentation](https://kind.sigs.k8s.io/docs/user/quick-start/#configuring-your-kind-cluster) for all options supported by kind. + +Now specify either `--kind-config` or `kindConfig` in your configuration file: + +```bash +kubectl kuttl test --kind-config=kind.yaml +``` + +*Note*: Once the tests have been completed, the test harness will collect the kind cluster's logs and then delete it, unless `--skip-cluster-delete` has been set. + +## Mocked Control Plane + +The above environments are great for end to end testing, however, for integration test use-cases it may be unnecessary to create actual pods or other resources. This can make the tests a lot more flaky or slow than they need to be. + +To write integration tests using the KUTTL test harness, it is possible to start a mocked control plane that starts only the Kubernetes API server and etcd. In this environment, objects can be created and operated on by custom controllers, however, there is no scheduler, nodes, or built-in controllers. This means that pods will never run and built-in types, such as, deployments cannot create pods. + +Kubernetes controllers can be added to this environment by using the TestSuite configuration command in order to start the controller: + +``` +commands: + - command: ./bin/manager + background: true +``` + +To start the mocked control plane, specify either `--start-control-plane` on the CLI or `startControlPlane` in the configuration file: + +```bash +kubectl kuttl test --start-control-plane +``` + +## Environment Setup + +Before running a test suite, it may be necessary to setup the Kubernetes cluster - typically, either installing required services or custom resource definitions. + +Your `kuttl-test.yaml` can specify the settings needed to setup the cluster: + +```yaml +apiVersion: kuttl.dev/v1beta1 +kind: TestSuite +startControlPlane: true +testDirs: +- tests/e2e/ +manifestDirs: +- tests/manifests/ +crdDir: tests/crds/ +commands: + - command: kubectl apply -f https://raw.githubusercontent.com/kudobuilder/kudo/master/docs/deployment/10-crds.yaml +``` + +The above configuration would start kind, install all of the CRDs in `tests/crds/`, and run all of the commands defined in `kubectl` before running the tests in `testDirs`. + +See the [configuration reference](reference.md#testsuite) for documentation on configuring test suites. + +### Starting a Kubernetes Controller + +In some test suites, it may be useful to have a controller running. To start a controller, add a configuration as a command in the TestSuite configuration file `kuttl-test.yaml`: + +For a KUDO, an example of deploying an previously released controller would look like: + +``` +commands: + - command: kubectl kudo init --wait +``` + +The KUDO CLI has a readiness watch on the installation of the KUDO manager. When it exits, the KUDO manager is ready. + +Another commonly explain is the starting of a manager that is still in development. The assumption of the code snippet below is that a `make manager` or Makefile target generated a manager in the `bin` folder. + +``` +commands: + - command: ./bin/manager + background: true +``` + +## KUTTL Mode of Testing in a Cluster + +KUTTL `test` is designed to function in 2 distinct modes managed by the use of the `--namespace` flag. + +1. By default, KUTTL will create a namespace, run a series of steps defined by a test, then delete the namespace. It will create a namespace for each test running in namespace isolation. Since, KUTTL owns the namespace, it deletes it as part of cleanup. + +1. When `--namespace` specifies a namespace, it is expected that the namespace exists. KUTTL in this mode, does **NOT** create or delete the namespace. All tests are run and share this namespace by default. If the namespace does NOT exist, the test fails. + +### Single Namespace Testing + +When running with the `--namespace`, there are potential consequences which are very important to understand. Normally when KUTTL is in the "apply" phase, if an object doesn't exist, it is created. If it does exist, it is merge patch updated. When creating a series of tests which do NOT share a namespace, potentially the same object is referenced in multiple tests. Those objects are separated by namespace and are auto-cleaned up by the deleting of the namespace. When running in the same namespace, this cleanup does NOT happen. It is the responsibility of the test designers to delete the objects pre- or post-test. This results in TestSuites designed to run in single namespace can be run in the default multi-namespace mode, but it is possible the reverse isn't true. More care needs to be taken in single namespace testing for pre/post test management. + +It is worth noting that extra care noted above is necessary for the "happy path". IF a test fails and does not clean up properly, some future test (during this testsuite) may be affected. For these reasons, running parallel tests for single namespace testing could also run into challenges and is not recommended. + +## Permissions / RBAC Rules + +KUTTL was initially designed to "own" a cluster for testing. In its default mode, it needs to be able to create and delete namespaces, as well as create/update/view kubernetes objects in that namespace. The RBAC needs in this mode include: + +1. POST, GET, LIST, PUT, PATCH, DELETE on namespace and the objects in that namespace. +1. GET, LIST events + +It is possible to turn off events with the `--suppress-log=events`. This removes the need to GET or LIST events. + +When running in single namespace testing mode, no permissions are needed for namespaces, reducing permissions to events. In this mode, it is possible to remove KUTTLs access needs by using the `--suppress-log=events`. In this mode, you will need access in the explicitly provided namespace to create, update and delete kubernetes objects defined in the test. + +**NOTE:** This defined permissions are for KUTTL itself and do NOT take in account the test that kuttl is running. It is possible for the test to create a namespace which is considered outside the KUTTL permission needs. diff --git a/docs/testing/tips.md b/docs/testing/tips.md new file mode 100644 index 00000000..342e57f6 --- /dev/null +++ b/docs/testing/tips.md @@ -0,0 +1,143 @@ +# Tips and Tricks + +This document contains some tips and gotchas that can be helpful when writing tests. + +## Loading Built Images Into KIND + +When KIND clusters are started, you may want to load an image that has not been pushed into the registry. To do this, you can use the `kindContainers` setting on your `TestSuite`. + +For example: + +```sh +docker build -t myimage . +``` + +And then in the TestSuite, set: + +```yaml +apiVersion: kuttl.dev/v1beta1 +kind: TestSuite +startKIND: true +kindContainers: +- myimage +``` + +When the KIND cluster is launched, the image will be loaded into it. + +## Kubernetes Events + +Kubernetes events are regular Kubernetes objects and can be asserted on just like any other object: + +```yaml +apiVersion: v1 +kind: Event +reason: Started +source: + component: kubelet +involvedObject: + apiVersion: v1 + kind: Pod + name: my-pod +``` + +## Custom Resource Definitions + +New Custom Resource Definitions are not immediately available for use in the Kubernetes API until the Kubernetes API has acknowledged them. + +If a Custom Resource Definition is being defined inside of a test step, be sure to to wait for the `CustomResourceDefinition` object to appear. + +For example, given this Custom Resource Definition in `tests/e2e/crd-test/00-crd.yaml`: + +```yaml +apiVersion: apiextensions.k8s.io/v1beta1 +kind: CustomResourceDefinition +metadata: + name: mycrds.mycrd.k8s.io +spec: + group: mycrd.k8s.io + version: v1alpha1 + names: + kind: MyCRD + listKind: MyCRDList + plural: mycrds + singular: mycrd + scope: Namespaced +``` + +Create the following assert `tests/e2e/crd-test/00-assert.yaml`: + +```yaml +apiVersion: apiextensions.k8s.io/v1beta1 +kind: CustomResourceDefinition +metadata: + name: mycrds.mycrd.k8s.io +status: + acceptedNames: + kind: MyCRD + listKind: MyCRDList + plural: mycrds + singular: mycrd + storedVersions: + - v1alpha1 +``` + +And then the CRD can be used in subsequent steps, `tests/e2e/crd-test/01-use.yaml`: + +```yaml +apiVersion: mycrd.k8s.io/v1alpha1 +kind: MyCRD +spec: + test: test +``` + +Note that CRDs created via the `crdDir` test suite configuration are available for use immediately and do not require an assert like this. + +## Helm testing + +You can test a Helm chart by installing it in either a test step or your test suite: + +```yaml +apiVersion: kuttl.dev/v1beta1 +kind: TestSuite +commands: +- command: kubectl create serviceaccount -n kube-system tiller + ignoreFailure: true +- command: kubectl create clusterrolebinding tiller --clusterrole=cluster-admin --serviceaccount=kube-system:tiller + ignoreFailure: true +- command: helm init --wait --service-account tiller +- command: helm delete --purge memcached + ignoreFailure: true +- command: helm install --replace --namespace memcached --name nginx stable/memcached +testDirs: +- ./test/integration +startKIND: true +kindNodeCache: true +``` + +## Image caching in kind + +By default, [kind](https://kind.sigs.k8s.io/) does not persist its containerd directory, meaning that on every test run you will have to download all of the images defined in the tests. However, the kuttl test harness supports creating a named Docker volume for each node specified in the kind configuration (or the default node if no nodes or configuration are specified) that will be used for each test run: + +```yaml +apiVersion: kuttl.dev/v1beta1 +kind: TestSuite +startKIND: true +kindNodeCache: true +testDirs: +- ./test/integration +``` + +The first time you run the tests, the nodes will download the images, but subsequent runs will used the cached images. + +## IDE completion for kuttl configuration files + +While there is no currently available K8S controller to handle the kuttl configuration files, +the [kuttl CRD definitions](https://github.com/kudobuilder/kuttl/blob/main/crds/) may be handy for kuttl users to leverage coding assistance for kuttl configuration files in +their favorite IDE. + +For intellij IDEA, see [instructions](https://www.jetbrains.com/help/idea/kubernetes.html#crd) for on how to load the CRD files either from: +- a local clone on your desktop +- remote github raw url pointing to the kuttl repository +- from a K8S cluster where you'd register the CRDs (by running `kubectl apply -f `) + +Screenshots in [PR #376](https://github.com/kudobuilder/kuttl/pull/376) \ No newline at end of file diff --git a/docs/what-is-kuttl.md b/docs/what-is-kuttl.md new file mode 100644 index 00000000..6cfd82cd --- /dev/null +++ b/docs/what-is-kuttl.md @@ -0,0 +1,23 @@ +# What is KUTTL + +## Overview + +The KUbernetes Test TooL (KUTTL) provides a declarative approach to testing production-grade Kubernetes [operators](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/). + +It provides a way to inject an operator (subject under test) during the TestSuite setup and allows tests to be standard YAML files. Test assertions are often partial YAML documents which assert the state defined is true. + +It is also possible to have KUTTL automate the setup of a cluster. + +## Motivation + +Testing Kubernetes operators is not easy. As the KUDO team was building a "declarative" Kubernetes operator, it just made sense to create a declarative way to test as well. The motivation is to leverage the existing Kubernetes eco-system for resource management (YAMLs) in a way to **setup** a test and as well as a way to **assert** state within the cluster. + +## When would you use KUTTL + +The testing eco-system is vast and includes at a minimum low level unit tests, integration tests and end-to-end testing. KUTTL is built to support some kubernetes integration test scenarios and is most valuable as an end-to-end (e2e) test harness. + +KUTTL is great when you want to: + +* Provide tests against your Custom Resource Definitions (CRDs) +* Inject a controller and assert states in a running cluster +* Test a set of TestSuites against multiple implementations and multiple versions of Kubernetes clusters.