feat: manage database migration #812

Ubisoft-potato · 2024-02-09T03:52:59Z

Fixes #685

Install atlas

We should install atlas first by running this command:

make local-deps

Migration workflow

Create migration file from the dev database. [1]
Explanation: This commmand will create migration files by comparing the migration file directory and dev database. For Example: when the migration directory is empty, after execute this command will creste the migration files that creates all tables.

Run this command will need Docker, beacause atlas need an ephemeral local Docker
container to create the migration file.

The migration files directory is migration/mysql.

Check the migration files. [1]
After create migration files, the directory will looks like the following:

migration/
├── Dockerfile
└── mysql
 ├── 20240311022556_initialization.sql
 ├── 20240311024017_add_c.sql
 └── atlas.sum

Suppose we are currently at version 20240311022556, so we will check the next migration file, which is version 20240311024017.
After executing this command, it will display the migration process for the database.

make apply-migration USER=bucketeer PASS=bucketeer HOST=localhost PORT=3306 DB=bucketeer
atlas migrate apply \
             --dir file://migration/mysql \
             --url mysql://bucketeer:bucketeer@localhost:3306/bucketeer \
     --dry-run
Migrating to version 20240311024017 from 20240311022556 (1 migrations in total):

-- migrating version 20240311024017
 -> ALTER DATABASE `bucketeer` CHARSET utf8 COLLATE utf8_general_ci;
 -> ALTER TABLE `bucketeer`.`admin_audit_log` ADD COLUMN `test_c` int NULL;
-- ok (4.198µs)

-------------------------
-- 6.229579ms
-- 1 migration
-- 2 sql statements

Merge the PR that contains the migration files. [1]
Triger Github Action to build the migration image. [1]

Helm Hook

As we need to do migration before install or upgrade Bucketeer Helm Charts, so we will add a pre-install and pre-upgrade Helm Hook to do migration process using the migration image that we build. [1] [2]

Additionally, we have to set the migration database URL.

If this Helm Hook Job fails, it will block the entire installation until it succeeds.

Other

If we need to use atlas migrattion in our existing Bucketeer cluster, we need to set the current version for atlas, because atlas will apply migration files from the first to last.
Additionally, atlas will save the migration version in the database table: atlas_schema_revisions.

For example, suppose our database version is at 20240311022556, we can use the following command to set the version for atlas:

make atlas-set-version VERSION=20240311022556 USER=bucketeer PASS=bucketeer HOST=localhost PORT=3306 DB=bucketeer
atlas migrate set 20240311022556 \
        --dir file://migration/mysql \
        --url mysql://bucketeer:bucketeer@localhost:3306/bucketeer
Current version is 20240311022556 (1 set):

  + 20240311022556 (initialization)

After set atlas version, we wll find atlas_schema_revisions table in database.

kentakozuka · 2024-02-28T00:21:19Z

@Ubisoft-potato migration pod has "helm.sh/hook": post-install, post-upgrade annotation. This means that the migration is executed after all k8s resources are updated, does it?
This causes errors because services of the new version expect the new schema.

IMO, it would be better to run migration before helm upgrades.

Ubisoft-potato · 2024-02-29T02:06:11Z

@Ubisoft-potato migration pod has "helm.sh/hook": post-install, post-upgrade annotation. This means that the migration is executed after all k8s resources are updated, does it? This causes errors because services of the new version expect the new schema.

IMO, it would be better to run migration before helm upgrades.

@kentakozuka Currently, the database migration API is implemented in the Batch service, so we can only wait for the Batch service to be deployed and then send the migration request to it.So I am using "helm.sh/hook": post-install, post-upgrade.

Let me check if we can make sure Batch service chart could be deployed first when we install or upgrade Bucketeer Helm Charts.

cre8ivejp · 2024-03-07T04:30:56Z

@Ubisoft-potato, as we spoke, we might achieve our goal of upgrading new schemas without causing downtime by running the migration process outside the Bucketeer app. We could use a script or container deployed temporarily and terminated after upgrading.

This guide seems a good solution to our problem.
https://atlasgo.io/guides/deploying/helm#using-helm-lifecycle-hooks
https://atlasgo.io/guides/deploying/k8s-init-container

Ubisoft-potato · 2024-03-07T04:38:25Z

@Ubisoft-potato, as we spoke, we might achieve our goal of upgrading new schemas without causing downtime by running the migration process outside the Bucketeer app. We could use a script or container deployed temporarily and terminated after upgrading.

This guide seems a good solution to our problem. https://atlasgo.io/guides/deploying/helm#using-helm-lifecycle-hooks

@cre8ivejp Let's eliminate migration in the batch service.

First, I will create our GitHub workflow for creating migration images using atlasgo. Then, I will add a pre-install helm hook for Bucketeer Charts.

IMO, it would be better to run migration before helm upgrades.

@kentakozuka Please check the new solution here.

kentakozuka · 2024-03-08T01:17:22Z

@Ubisoft-potato

First, I will create our GitHub workflow for creating migration images using atlasgo. Then, I will add a pre-install helm hook for Bucketeer Charts.

Sounds good to me!

kentakozuka

@Ubisoft-potato Thanks!
How do we rollback a migration if the migration succeeded but we want to revert it?

kentakozuka · 2024-04-03T01:27:59Z

Makefile

+  		--dir file://migration/mysql \
+  		--to mysql://${USER}:${PASS}@${HOST}:${PORT}/${DB} \
+  		--dev-url docker://mysql/8


Replace spaces with tabs.

kentakozuka · 2024-04-03T01:28:17Z

Makefile

+  		--dir file://migration/mysql \
+  		--url mysql://${USER}:${PASS}@${HOST}:${PORT}/${DB} \


Replace spaces with tabs.

kentakozuka · 2024-04-03T05:19:20Z

Makefile

+.PHONY: install-atlas
+install-atlas:
+	curl -sSf https://atlasgo.sh | sh


Please add these lines to local-deps.

Ubisoft-potato · 2024-04-05T02:02:44Z

@Ubisoft-potato Thanks! How do we rollback a migration if the migration succeeded but we want to revert it?

@kentakozuka This PR does not include the rollback migration implementation. We will address it in a new PR due to the complexity of the atlas down migration.

kentakozuka

LGTM 🚀

Signed-off-by: Alessandro Yuichi Okimoto <[email protected]>

cre8ivejp · 2024-06-19T08:44:08Z

.github/workflows/push-image.yaml

@@ -245,7 +267,7 @@ jobs:
          token: ${{ secrets.REPO_ACCESS_PAT }}
          workflow: dev-deploy.yaml
          ref: master
-          inputs: '{"ref": "${{ needs.setup.outputs.REF }}", "image_tag": "${{ needs.setup.outputs.IMAGE_TAG }}", "committer": "${{ github.event.head_commit.author.name }}", "commit_message": "${{ needs.setup.outputs.COMMIT_MESSAGE }}", "commit_url": "${{ github.event.head_commit.url }}", "skip_e2e": "${{ github.event.inputs.skip_e2e }}", "skip_sdk_e2e": "${{ github.event.inputs.skip_sdk_e2e }}", "skip_prd_pr": "${{ github.event.inputs.skip_prd_pr }}"}'
+          inputs: '{"ref": "${{ needs.setup.outputs.REF }}", "image_tag": "${{ needs.setup.outputs.IMAGE_TAG }}", "committer": "${{ github.event.head_commit.author.name }}", "commit_message": "${{ needs.setup.outputs.COMMIT_MESSAGE }}", "commit_url": "${{ github.event.head_commit.url }}", "skip_e2e": "${{ github.event.inputs.skip_e2e }}", "skip_sdk_e2e": "${{ github.event.inputs.skip_sdk_e2e }}", "skip_prd_pr": "${{ github.event.inputs.skip_prd_pr }}", "migration_required": "${{ needs.setup.outputs.MIGRATION_REQUIRED }}"}'


When deploying, we will need the migration_required to update the migration container image tag in the dev environment.

cre8ivejp · 2024-06-19T08:46:34Z

.github/workflows/push-image.yaml

@@ -98,13 +106,17 @@ jobs:
            IMAGE=`./tools/build/show-image-name.sh $APP`
            docker build -f Dockerfile-app-$APP -t bucketeer-$IMAGE:${{ needs.setup.outputs.IMAGE_TAG }} --load .
          done
+      - name: Build and push the Migration image
+        if: needs.setup.outputs.MIGRATION_REQUIRED == 'true'


It will only build and push the image when the migration directory changes.

cre8ivejp · 2024-06-19T08:49:20Z

manifests/bucketeer/values.yaml

+migration:
+  image:
+    repository: ghcr.io/bucketeer-io/bucketeer-migration
+    tag: v0.4.5 # x-release-please-version


Set the default tag and the comment so the Release Please can update the tag when releasing.

cre8ivejp · 2024-06-19T08:55:01Z

.github/workflows/push-image.yaml

+      - name:  Store the migration_required value to output
+        uses: dorny/paths-filter@de90cc6fb38fc0963ad72b210f1f284cd68cea36 # v3.0.2
+        id: changes
+        with:
+          filters: |
+            migration:
+              - 'migration/**'


I'm using the dorny/paths-filter action, which is great for checking for changes in directories or files.

Signed-off-by: Alessandro Yuichi Okimoto <[email protected]>

cre8ivejp · 2024-06-26T08:19:48Z

Closing this in favor of #1060

Ubisoft-potato force-pushed the feature-migration branch 2 times, most recently from f94c263 to 8ee18f2 Compare February 9, 2024 05:19

Ubisoft-potato marked this pull request as ready for review February 9, 2024 06:46

Ubisoft-potato requested review from cre8ivejp and kentakozuka as code owners February 9, 2024 06:46

Ubisoft-potato force-pushed the feature-migration branch 2 times, most recently from 9022b58 to f1bef92 Compare February 21, 2024 02:41

Ubisoft-potato force-pushed the feature-migration branch from a21ec3c to 25260b3 Compare February 28, 2024 08:47

cre8ivejp mentioned this pull request Feb 29, 2024

chore: enable version-specific db migration #453

Closed

Ubisoft-potato force-pushed the feature-migration branch from f4a3257 to 3d74ca9 Compare March 11, 2024 04:23

Ubisoft-potato force-pushed the feature-migration branch from 83cbfa0 to 1befca2 Compare March 19, 2024 02:18

kentakozuka reviewed Apr 3, 2024

View reviewed changes

Ubisoft-potato force-pushed the feature-migration branch from 1befca2 to 4628551 Compare April 5, 2024 13:01

kentakozuka previously approved these changes Apr 10, 2024

View reviewed changes

Ubisoft-potato dismissed kentakozuka’s stale review via acadfd7 April 15, 2024 02:39

Ubisoft-potato force-pushed the feature-migration branch from fa65d5f to acadfd7 Compare April 15, 2024 02:39

cre8ivejp marked this pull request as draft June 12, 2024 06:57

Ubisoft-potato force-pushed the feature-migration branch 2 times, most recently from 43fc76a to bf7fca3 Compare June 19, 2024 04:33

Ubisoft-potato added 5 commits June 19, 2024 17:27

feat: database migration

1b05bd4

feat: batch service check system admin

5d0b7e9

feat: add initial migration file

502429d

chore: remove migration from backend

5f634ca

chore: update repo

0034104

Ubisoft-potato and others added 14 commits June 19, 2024 17:27

chore: add helm migration hook

9069d43

chore: gen proto file

f24def8

chore: remove migration from batch

c7a69eb

chore: add atlas migration tool

80d2d9a

chore: tidy deps

bc8b2ce

fix: golang lint

4c5b051

chore: add migration files

47a688d

chore: remove batch migration manifest

8578ae1

chore: remove batch migration manifest

c5217d9

chore: go tidy mod

a1cc983

chore: use tab

ce966a3

chore: go mod

abdb3c0

chore: add oauth config

d9d4c1a

chore: update init migration file

5d6ea8b

Signed-off-by: Alessandro Yuichi Okimoto <[email protected]>

cre8ivejp force-pushed the feature-migration branch from 4f9907a to 09d6fa7 Compare June 19, 2024 08:41

ci: configure migration step in the push-image workflow

7e289b8

Signed-off-by: Alessandro Yuichi Okimoto <[email protected]>

cre8ivejp force-pushed the feature-migration branch from 09d6fa7 to 7e289b8 Compare June 19, 2024 08:42

cre8ivejp reviewed Jun 19, 2024

View reviewed changes

chore: add imagePullSecrets for private registries

74dc8ad

Signed-off-by: Alessandro Yuichi Okimoto <[email protected]>

cre8ivejp force-pushed the feature-migration branch from 0d72467 to 74dc8ad Compare June 19, 2024 10:21

cre8ivejp closed this Jun 26, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: manage database migration #812

feat: manage database migration #812

Ubisoft-potato commented Feb 9, 2024 •

edited

Loading

kentakozuka commented Feb 28, 2024

Ubisoft-potato commented Feb 29, 2024

cre8ivejp commented Mar 7, 2024 •

edited

Loading

Ubisoft-potato commented Mar 7, 2024 •

edited

Loading

kentakozuka commented Mar 8, 2024

kentakozuka left a comment

kentakozuka Apr 3, 2024

kentakozuka Apr 3, 2024

kentakozuka Apr 3, 2024

Ubisoft-potato commented Apr 5, 2024 •

edited

Loading

kentakozuka left a comment

cre8ivejp Jun 19, 2024

cre8ivejp Jun 19, 2024

cre8ivejp Jun 19, 2024

cre8ivejp Jun 19, 2024

cre8ivejp commented Jun 26, 2024

		--dir file://migration/mysql \
		--url mysql://${USER}:${PASS}@${HOST}:${PORT}/${DB} \

feat: manage database migration #812

feat: manage database migration #812

Conversation

Ubisoft-potato commented Feb 9, 2024 • edited Loading

Install atlas

Migration workflow

Helm Hook

Other

kentakozuka commented Feb 28, 2024

Ubisoft-potato commented Feb 29, 2024

cre8ivejp commented Mar 7, 2024 • edited Loading

Ubisoft-potato commented Mar 7, 2024 • edited Loading

kentakozuka commented Mar 8, 2024

kentakozuka left a comment

Choose a reason for hiding this comment

kentakozuka Apr 3, 2024

Choose a reason for hiding this comment

kentakozuka Apr 3, 2024

Choose a reason for hiding this comment

kentakozuka Apr 3, 2024

Choose a reason for hiding this comment

Ubisoft-potato commented Apr 5, 2024 • edited Loading

kentakozuka left a comment

Choose a reason for hiding this comment

cre8ivejp Jun 19, 2024

Choose a reason for hiding this comment

cre8ivejp Jun 19, 2024

Choose a reason for hiding this comment

cre8ivejp Jun 19, 2024

Choose a reason for hiding this comment

cre8ivejp Jun 19, 2024

Choose a reason for hiding this comment

cre8ivejp commented Jun 26, 2024

Ubisoft-potato commented Feb 9, 2024 •

edited

Loading

cre8ivejp commented Mar 7, 2024 •

edited

Loading

Ubisoft-potato commented Mar 7, 2024 •

edited

Loading

Ubisoft-potato commented Apr 5, 2024 •

edited

Loading