Skip to content
This repository has been archived by the owner on Oct 22, 2021. It is now read-only.

Helm Secret Management

Jump to bottom
viovanov edited this page May 2, 2018 · 3 revisions

Sample charts are available here.

Use cases

  1. User must be able to individually set secrets

This is simple and works as expected using typical helm templates.

  1. When user changes a secret, roles that use it should restart
helm upgrade --set secrets.mypassword=newpass mysecrets .

This command works when using the method described here: https://github.com/kubernetes/helm/blob/master/docs/charts_tips_and_tricks.md#automatically-roll-deployments-when-configmaps-or-secrets-change

  1. Must be able to generate secrets; the generated secrets should be used if they have not been set by the user

IMPORTANT: the secret generation job in the sample is not smart and only works with the "secrets" namespace

This is possible with an appropriate service account. Our current secret generation job does this. The sample uses kubectl to make the process easier to follow.

  1. When upgrading/installing and new secrets are created, roles do not start until all the secrets they need have been generated

For installation, if a secret is not set (null or empty ""), it's generated by the image and roles don't start until the secret is created. Same behavior as seen today - CreateContainerConfigError until the secret generation job completes.

  1. User should be able to replace a generated secret with a manual secret

When a user wants to manually set a password (after it's been generated), they run the following command, and the manual secret will be used:

helm upgrade --set secrets.mypassword=newpass mysecrets .

IMPORTANT: in the sample, after upgrading to a new version and changing v3pass to a manual value, both deployments are rotated. this happens because the checksum is set for all of the secrets file; this could be eleviated by using one file/kube secret for each secret.

We assume there are no race conditions between subsequent updates of a manual secret (i.e. if I change a manual secret, the pods that use it won't be restarted and lookup the old value), because there's an order to how helm installs resources, detailed here: https://docs.helm.sh/developing_charts/#operational-aspects-of-using-dependencies

  1. User should be able to switch back to using generated secrets

The user sets the value of the secret to an empty value "" with an upgrade command:

helm upgrade --set secrets.mypassword="" mysecrets .
  1. When a non-secret variable is set, secrets are not rotated, and only the roles depending on the variable are restarted

Using the following command, a user can set a value and only the roles referencing it are restarted:

helm upgrade --set env.myvalue="newvalue" mysecrets .
  1. When an upgrade to a new release happens, the roles wait for new secrets (that did not exist in the previous version) to be created, and old generated secrets are not overwritten.

IMPORTANT: In the sample, the code to retrieve an existing secret value and use it for the new iteration is ommited. For the secret generation job to be able to read old secrets, it's required that the old version of the chart is persisted in a config map.

This works by using {{ .Chart.Version }} in the name of the generated secret. The job name contains {{ .Chart.Version }} and {{ .Values.kube.generated_secrets_counter }} in order to support generated secret rotation. Jobs are correctly restarted and wait for the new secret to be available. This does not interfere with #7, as the chart version is not changed when a helm upgrade with the same version occurs.

  1. User should be able to rotate generated secrets

This is possible by using a helm variable that is manually incremented when rotation is wanted, e.g. kube.generated_secrets_counter: 1. The counter must be a positive integer.

The {{ .Values.kube.generated_secrets_counter }} is included in the generated secret name. This will cause helm templates for pods that reference the secret to change, thus restarting. The secrets job will override mutable secrets because it can detect the generation counter has changed from the previous secret name.

All roles will wait because the new secret does not yet exist.

  1. Secrets that are marked as immutable cannot be set by the user, are always generated and are not rotated

Commands to use for testing

IMPORTANT: Please note that proper upgrades must always involve working with a values file, not using the --set flag, since these don't persist from one upgrade to the next. We're using --set in these commands for convenience only, and only for this specific test script. Proper test cases should persist these values in a values file. The need to work with a values file needs to be documented.

# Use `k logs :hello` or `k logs :value` to see the secrets/configs in the deployments


cd ./secrets-spec

# Install and test usecase #3 and #4 - secrets are generated and roles wait
helm install . --namespace secrets --name mysecrets

# Test usecase #2 and #5 - set a manual secret
helm upgrade --set secrets.mypassword=newpass mysecrets .

# Test usecase #6 - go back to a generated secret
helm upgrade --set secrets.mypassword="" mysecrets .

# Test usecase #7 - set a value and notice only the value container is restarted
helm upgrade --set env.myvalue="newvalue" mysecrets .

# Test usecase #8 - upgrade to a new version and see new secrets being created and roles waiting for them
cd ../secrets-spec-v3
helm upgrade mysecrets .

# Test usecase #9 - rotate all generated secrets
helm upgrade --set kube.generated_secrets_counter=2 mysecrets .

What has not been covered

Deleting and cleaning up secrets from one version to the next.

Clone this wiki locally