feat(helm)!: add a runMigrationInPreDeployment value allowing to move chartreuse migration to pre-deployment instead of post-deployment, false by default.

Author: desaintmartin

.dockerignore

@@ -0,0 +1,8 @@
+bin
+documentation
+include
+kubernetes
+lib
+other
+venv
+.*

.gitignore

@@ -0,0 +1 @@
+*.tgz

CHANGELOG.md

@@ -2,14 +2,17 @@
 ## BREAKING CHANGES
 - helm: move most alembic/eslembic parameters to eslembic and alembic prefix.
 - Rename most environment variables used by chartreuse.
+- Remove pre-deployment job that stop-pods before deployment, since helm 3 will re-start them anyway.
 ## Features
+- Chartreuse: add a runMigrationInPreDeployment value allowing to move chartreuse migration to pre-deployment instead of post-deployment, false by default. Note that in such case, a deployment failure may cause new database schema version but old code to be on the environment (you may want to disable helmDefaults.atomic parameter in helmfile.yaml of your project)
+- Chartreuse: Allow to keep pods started and not do stop-pods.
+- alembic: allow to give additional parameters to alembic.
+- eslembic: Split eslembic upgrade to eslembic upgrade and eslembic migrate as a post job
+- eslembic: allow to enable eslembic upgrade through `eslembic.upgrade.enabled` value
+- helm: Add `additionalEnvironmentVariables` parameter to inject envvars into Chartreuse Pods.
 - helm: Add support for built-in secret.
 - helm: delete all roles, serviceaccounts, secrets, configmap after success
-- Allow to keep pods started and not do stop-pods.
-- alembic: allow to give additional parameters to alembic.
-- eslembic: Split eslembic upgrade to eslembic migrate and eslembic upgrade as a post job
-- eslembic: allow to disable/enable eslembic upgrade
-## Fixes
+- helm: delete post-rollback job that is useless with helm 3
 - Upgrade to eslembic v6.x.x
 
 # 1.1.0 (2020-03-10)

README.md

@@ -2,7 +2,7 @@
 
 Chartreuse is a wrapper around [Alembic](https://alembic.sqlalchemy.org) and [Eslembic](https://gitlab.cayzn.com/wiremind/commons/eslembic) to ease,
  detect and automate migrations on deployed applications.
- 
+
 Chartreuse is made to work as Helm hooks. You need to use Chartreuse a a sub-chart of your project.
 
 # Install
@@ -27,9 +27,17 @@ upgrade (starting from a version `V`) but the same scenario
 applies to applications when installing them for the first time
 (consider version `V` to be scratch).
 
-- The diagram has been drawn using the free online software https://draw.io, the 
+- The diagram has been drawn using the free online software https://draw.io, the
 source code is located at `doc/chartreuse_sd.xml`, feel free
 to correct it or make it more understandable.
 
 - In the end of an install/upgrade, whatever its state: succeeded or failed, make sure
-that the pods were scaled up as expected, if it isn't the case, this should be done manually.
+that the pods were scaled up as expected, if it isn't the case, this should be done manually.
+
+# Test
+
+There are three kind of tests:
+
+- Unit tests
+- Integration tests: allows to run in a real environment, but still control chartreuse from the inside
+- blackbox test: deploy a real Helm Release and test if databases are migrated. Requires the PYPI_PASSWORD_READONLY envvar to be set in order to fetch required eggs to build the test Docker image.

VERSION

@@ -1 +1 @@
-2.0.0-dev0
+2.0.0-dev13

helm-chart/chartreuse/Chart.yaml

@@ -1,4 +1,5 @@
 apiVersion: v1
 name: chartreuse
-version: 2.0.0-dev0
+# Note: when setting a development version, please explicitely specify it in e2e test helm chart, otherwise just put *.
+version: 2.0.0
 description: Migrate databases with ease

helm-chart/chartreuse/templates/_helpers.tpl

@@ -54,3 +54,53 @@ Create the name of the service account to use
     {{ default "default" .Values.serviceAccount.name }}
 {{- end -}}
 {{- end -}}
+
+
+{{- define "chartreuse.alembic.url" -}}
+{{ printf "postgresql://%s:%s@%s%s:5432/%s?sslmode=prefer" .Values.alembic.username .Values.alembic.password .Release.Name .Values.alembic.urlSuffix .Values.alembic.database }}
+{{- end -}}
+
+{{- define "chartreuse.eslembic.url" -}}
+{{ printf "http://%s%s:9200" .Release.Name .Values.eslembic.urlSuffix }}
+{{- end -}}
+
+
+{{- define "chartreuse.annotations" -}}
+{{- if .Values.runMigrationInPreDeployment }}
+{{- if .Release.IsInstall }}
+# No hook: we deploy this job during the initial install, as part of the Helm Release
+{{- end }}
+{{- if .Release.IsUpgrade }}
+# Should be run in pre-upgrade only (not pre-install)
+"helm.sh/hook": pre-upgrade
+"helm.sh/hook-weight": "1"
+"helm.sh/hook-delete-policy": "before-hook-creation,hook-succeeded"
+{{- end }}
+{{- else }}
+# Should be run in post-install,post-upgrade wherever it is install or upgrade.
+"helm.sh/hook": post-install,post-upgrade
+"helm.sh/hook-weight": "1"
+"helm.sh/hook-delete-policy": "before-hook-creation,hook-succeeded"
+{{- end }}
+{{- end -}}
+
+{{- define "chartreuse.annotations.ephemeral" -}}
+{{- if .Values.runMigrationInPreDeployment }}
+"helm.sh/hook": pre-upgrade
+{{- else }}
+"helm.sh/hook": post-install,post-upgrade
+{{- end }}
+"helm.sh/hook-weight": "0"
+"helm.sh/hook-delete-policy": "before-hook-creation,hook-succeeded,hook-failed"
+{{- end -}}
+
+# Adds suffix -ephemeral if it is a helm hook
+{{- define "chartreuse.hook.suffix" -}}
+{{- if .Values.runMigrationInPreDeployment -}}
+{{- if .Release.IsUpgrade -}}
+-ephemeral
+{{- end -}}
+{{- else -}}
+-ephemeral
+{{- end -}}
+{{- end -}}

helm-chart/chartreuse/templates/confimap-ephemeral.yaml

@@ -0,0 +1,42 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: {{ template "chartreuse.fullname" . }}-ephemeral
+  labels:
+{{ include "chartreuse.labels" . | indent 4 }}
+  annotations:
+{{ include "chartreuse.annotations.ephemeral" . | indent 4 }}
+data:
+  # Note: Chartreuse expects false values to be empty string (falsy value) and true values to be non-empty string.
+{{- if .Values.alembic.allowMigrationFromEmptyDatabase }}
+  CHARTREUSE_ALEMBIC_ALLOW_MIGRATION_FOR_EMPTY_DATABASE: "1"
+{{- else }}
+  CHARTREUSE_ALEMBIC_ALLOW_MIGRATION_FOR_EMPTY_DATABASE: ""
+{{- end }}
+
+{{- if .Values.eslembic.cleanIndex.enabled }}
+  CHARTREUSE_ESLEMBIC_ENABLE_CLEAN: "1"
+{{- else }}
+  CHARTREUSE_ESLEMBIC_ENABLE_CLEAN: ""
+{{- end }}
+
+{{- if .Values.eslembic.migrate.enabled }}
+  CHARTREUSE_ESLEMBIC_ENABLE_MIGRATE: "1"
+{{- else }}
+  CHARTREUSE_ESLEMBIC_ENABLE_MIGRATE: ""
+{{- end }}
+
+{{- if .Values.stopPods }}
+  CHARTREUSE_ENABLE_STOP_PODS: "1"
+{{- else }}
+  CHARTREUSE_ENABLE_STOP_PODS: ""
+{{- end }}
+
+  CHARTREUSE_MIGRATE_CONTAINER_IMAGE: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
+  CHARTREUSE_MIGRATE_IMAGE_PULL_SECRET: "{{ .Values.image.pullSecret }}"
+  CHARTREUSE_RELEASE_NAME: {{ .Release.Name }}
+  CHARTREUSE_ALEMBIC_ADDITIONAL_PARAMETERS: "{{ .Values.alembic.additionalParameters }}"
+
+{{- with .Values.additionalEnvironmentVariables }}
+{{ toYaml . | indent 2 }}
+{{- end }}

helm-chart/chartreuse/templates/confimap.yaml

@@ -4,13 +4,8 @@ metadata:
   name: {{ template "chartreuse.fullname" . }}
   labels:
 {{ include "chartreuse.labels" . | indent 4 }}
-  annotations:
-    "helm.sh/hook": pre-install,pre-upgrade,post-install,post-upgrade,post-rollback
-    "helm.sh/hook-weight": "0"
-    "helm.sh/hook-delete-policy": "before-hook-creation,hook-succeeded"
 data:
-  # Chartreuse expects false values to be empty string (falsy value) and true values to be non-empty string.
-
+  # Note: Chartreuse expects false values to be empty string (falsy value) and true values to be non-empty string.
 {{- if .Values.alembic.allowMigrationFromEmptyDatabase }}
   CHARTREUSE_ALEMBIC_ALLOW_MIGRATION_FOR_EMPTY_DATABASE: "1"
 {{- else }}
@@ -23,10 +18,10 @@ data:
   CHARTREUSE_ESLEMBIC_ENABLE_CLEAN: ""
 {{- end }}
 
-{{- if .Values.eslembic.upgrade.enabled }}
-  CHARTREUSE_ESLEMBIC_ENABLE_UPGRADE: "1"
+{{- if .Values.eslembic.migrate.enabled }}
+  CHARTREUSE_ESLEMBIC_ENABLE_MIGRATE: "1"
 {{- else }}
-  CHARTREUSE_ESLEMBIC_ENABLE_UPGRADE: ""
+  CHARTREUSE_ESLEMBIC_ENABLE_MIGRATE: ""
 {{- end }}
 
 {{- if .Values.stopPods }}
@@ -35,7 +30,11 @@ data:
   CHARTREUSE_ENABLE_STOP_PODS: ""
 {{- end }}
 
-  CHARTREUSE_POST_UPGRADE_CONTAINER_IMAGE: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
+  CHARTREUSE_MIGRATE_CONTAINER_IMAGE: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
+  CHARTREUSE_MIGRATE_IMAGE_PULL_SECRET: "{{ .Values.image.pullSecret }}"
   CHARTREUSE_RELEASE_NAME: {{ .Release.Name }}
+  CHARTREUSE_ALEMBIC_ADDITIONAL_PARAMETERS: "{{ .Values.alembic.additionalParameters }}"
 
-  CHARTREUSE_ALEMBIC_ADDITIONAL_PARAMETERS: {{ .Values.alembic.additionalParameters }}
+{{- with .Values.additionalEnvironmentVariables }}
+{{ toYaml . | indent 2 }}
+{{- end }}