infra migrate-from-legacy: Utilize migration checkpoints

Author: doshitan

docs/getting-started/migrating-from-legacy-template.md

@@ -8,14 +8,35 @@ Platform CLI you'll need to convert the old file into the new format.
 The Platform CLI provides commands for doing this migration, though the exact
 steps you need to take will vary depending on what templates you have installed.
 
-> [!IMPORTANT]
->
-> If you running a very old (pre-summer 2024) version of a template
-> (particularly `template-infra`), reach out to the platform team for some
-> guidance.
-
 ## template-infra
 
+The switch to Platform CLI happened with `v0.15.0`. If you are running a version
+earlier than this, you'll need to migrate things.
+
+One way to figure out what version of `template-infra` your project is using is
+to run, at the root of your project:
+
+```sh
+nava-platform infra info --template-uri gh:navapbc/template-infra .
+```
+
+Look for the "Closest upstream version" value. If it is "Unknown", reach out to
+the Platform team for guidance.
+
+If the value is pre-`v0.12.0`, you may want to approach the update in smaller
+steps than jumping directly to `v0.15.0`. You can use the Platform CLI to do
+these updates as well, see [Migrate in smaller
+steps](#migrate-in-smaller-steps).
+
+As always, read the [release
+notes](https://github.com/navapbc/template-infra/releases) for each version
+between your current one and your ultimate target. This process does not
+eliminate the need to apply the state changes/manual migration steps, it just
+updates the code. See [Version callouts](#version-callouts) below for some
+particular things to consider.
+
+### Migrate to latest
+
 To transform the old `.template-version` file into the new format, run:
 
 ```sh
@@ -26,7 +47,9 @@ This will result in a `.template-infra/` directory with a number of files inside
 of it. Check that the `app-<APP_NAME>.yml` files all correspond to proper
 applications. Remove any that don't and update the commit.
 
-Now perform the update, with:
+This gets your project into a state that Platform CLI can understand.
+
+Now perform the actual template update, with:
 
 ```sh
 nava-platform infra update .
@@ -46,6 +69,78 @@ nava-platform infra update-app --all .
 Likely you'll hit merge conflicts for each app as well, resolve those, commit,
 and move on to the next app, until you've done them all.
 
+See [the docs on updating in general](../updating.md) for more details on running
+updates.
+
+### Migrate in smaller steps
+
+This is similar to the previous section, so read that first.
+
+1. Run the `migrate-from-legacy` command as stated in previous section. This
+   gets you into the Platform CLI ecosystem.
+2. Then decide which version of `template-infra` you want to update to,
+   represented by `v0.x.x` in the following example:
+   ```sh
+   nava-platform infra update --version platform-cli-migration/v0.x.x .
+   ```
+3. Follow update guidance as discussed in previous section.
+4. Do steps 2-3 over and over, jumping versions as you see fit until you hit
+   `v0.15.0`.
+5. Once on `v0.15.0`, run a final update to get to the latest release (or to
+   whatever post-`v0.15.0` version you want):
+   ```sh
+   nava-platform infra update [--version vA.B.C] .
+   ```
+
+### Version callouts
+
+No substitute for reading the [release
+notes](https://github.com/navapbc/template-infra/releases), but here are a few
+points to consider when deciding what version to update to if you are
+significantly behind the latest:
+
+- A Feature Flags module, backed by AWS Evidently, was added in
+  [v0.5.0](https://github.com/navapbc/template-infra/releases/tag/v0.5.0) and
+  removed in
+  [v0.13.0](https://github.com/navapbc/template-infra/releases/tag/v0.13.0).
+    - If you are coming from pre-v0.5.0, you can delete the feature flag module
+      as you move past v0.5.0, or just ignore/don't change anything about it and
+      it will get cleaned up once you are post-v0.13.0.
+- [v0.9.0](https://github.com/navapbc/template-infra/releases/tag/v0.9.0) moved
+  account mapping to each environment config file, then
+  [v0.11.0](https://github.com/navapbc/template-infra/releases/tag/v0.11.0)
+  removed it from each environment config file and moved it to the network config.
+    - If you are pre-v0.9.0, you may want to consider jumping to v0.11.x+ to
+      avoid dealing with moving things multiple times.
+
+Misc. others:
+
+- [v0.11.0](https://github.com/navapbc/template-infra/releases/tag/v0.11.0)
+    - Starts pinning specific Terraform version in CI/CD
+- [v0.10.0](https://github.com/navapbc/template-infra/releases/tag/v0.10.0)
+    - DB changes: PostgreSQL version update to 16.2 and DB schema name hardcoded
+to `app`
+- [v0.9.0](https://github.com/navapbc/template-infra/releases/tag/v0.9.0)
+    - Requires Terraform 1.8.x (previous requirement was just >=1.4, more or less)
+    - Changes the way secrets are defined
+- [v0.7.0](https://github.com/navapbc/template-infra/releases/tag/v0.7.0)
+    - Minor state migration needed
+- [v0.6.0](https://github.com/navapbc/template-infra/releases/tag/v0.6.0)
+    - Networking changes likely requiring hours of downtime to apply
+
+### Post-migration
+
+After completing the migration, you may want to see what results from
+re-applying, more holistically, the latest (or your ultimate target) version of
+the template to the project:
+
+```sh
+nava-platform infra update --force [--version vA.B.C] .
+```
+
+This discards some of the "smart" logic of a regular update and might catch some
+things that were missed while trying to be smarter.
+
 ## Application templates
 
 These are historically less standard, so you'll have to provide a little more

docs/updating.md

@@ -24,10 +24,12 @@ Conflicts on Update](./avoiding-conflicts-on-update.md).
 nava-platform infra update .
 ```
 
-This can often run into merge conflicts that need resolved manually though. The
-tool will provide some guidance if this happens. But you can also approach the
-update in the separate pieces yourself, first updating the infrastructure base
-with:
+This will attempt to update the "base" template then each "app" instance in
+sequence. This can often run into merge conflicts that need resolved manually.
+The tool will provide some guidance if this happens.
+
+But you can also approach the update in the separate pieces yourself, first
+updating the infrastructure base with:
 
 ```sh
 nava-platform infra update-base .

nava/platform/cli/commands/infra/migrate_from_legacy_command.py

@@ -35,7 +35,9 @@ def _migrate_from_legacy(
         new_version_answers_file_name="base.yml",
         extra_answers=lambda _: (base_project_config_answers | {"template": "base"}),
     )
-    base_migrate.migrate_from_legacy(preserve_legacy_file=True, commit=commit)
+    base_migrate.migrate_from_legacy(
+        preserve_legacy_file=True, commit=commit, use_migration_tags=True
+    )
 
     for app_name in infra_project.app_names_possible:
         app_answers = {"app_name": app_name, "template": "app"}
@@ -48,7 +50,9 @@ def _migrate_from_legacy(
             new_version_answers_file_name=f"app-{app_name}.yml",
             extra_answers=lambda _: app_answers,  # noqa: B023
         )
-        app_migrate.migrate_from_legacy(preserve_legacy_file=True, commit=commit)
+        app_migrate.migrate_from_legacy(
+            preserve_legacy_file=True, commit=commit, use_migration_tags=True
+        )
 
     # remove the old file once we are done with it
     ctx.console.print(f"Deleting legacy file ({base_migrate.legacy_version_file_path()})")

nava/platform/projects/migrate_from_legacy_template.py

@@ -9,6 +9,7 @@
 from nava.platform.get_template_name_from_uri import get_template_name_from_uri
 from nava.platform.projects.project import Project
 from nava.platform.types import RelativePath
+from nava.platform.util.git import GitProject
 
 
 @dataclass
@@ -61,7 +62,12 @@ def answers_file_rel(self) -> RelativePath:
     def answers_file(self) -> Path:
         return self.project.dir / self.answers_file_rel()
 
-    def migrate_from_legacy(self, preserve_legacy_file: bool = False, commit: bool = False) -> None:
+    def migrate_from_legacy(
+        self,
+        preserve_legacy_file: bool = False,
+        commit: bool = False,
+        use_migration_tags: bool = False,
+    ) -> None:
         if not self.has_legacy_version_file:
             raise ValueError(
                 f"No legacy version file found (looking for {self.legacy_version_file_path()})."
@@ -74,10 +80,23 @@ def migrate_from_legacy(self, preserve_legacy_file: bool = False, commit: bool =
         if not self.project_state_dir().exists():
             self.project_state_dir().mkdir()
 
-        template_version = self.legacy_version_file_path().read_text()
-        short_version = template_version[:7]
+        template_version = self.legacy_version_file_path().read_text().strip()
+
+        ref = template_version
+        if use_migration_tags:
+            with GitProject.clone_if_necessary(self.origin_template_uri) as template_git:
+                ref, perfect_match = get_closest_migration_tag(template_git, template_version)
+
+            if not ref:
+                raise ValueError("Issue finding suitable migration point")
+
+            if not perfect_match:
+                self.ctx.console.warning.print(
+                    f"Couldn't find a perfect match to the current commit, using closest tagged version '{ref}' which is slightly older."
+                )
+
         common_answers = {
-            "_commit": short_version,
+            "_commit": ref,
             # Copier requires this to be set to a valid template path, and that template git project
             # needs to have _commit as a valid commit hash
             # If _src_path is not set, run_update will raise
@@ -111,3 +130,57 @@ def _extra_answers(self: Self) -> dict[str, str]:
             return self.extra_answers(self)
 
         return {}
+
+
+def get_closest_tag_before_commit(git: GitProject, commit: str) -> str:
+    """Find nearest tag before given commit."""
+    from nava.platform.cli.commands.infra.info_command import get_version
+
+    closest_tag = git.get_closest_tag(commit)
+    if not closest_tag:
+        raise Exception(f"Can't find closest tag for {commit}")
+
+    closest_version = get_version(closest_tag)
+    if not closest_version:
+        raise Exception(f"Can't determine version from tag {closest_tag}")
+
+    return "v" + closest_version.base_version
+
+
+MIGRATION_TAG_PREFIX = "platform-cli-migration/"
+
+
+def get_closest_migration_tag(git: GitProject, commit: str) -> tuple[str, bool]:
+    """Find nearest migration tag before given commit."""
+    from nava.platform.cli.commands.infra.info_command import get_version
+
+    closest_tag = get_closest_tag_before_commit(git, commit)
+    migration_tags = git.get_tags("--list", f"{MIGRATION_TAG_PREFIX}*")
+    if not migration_tags:
+        raise Exception("Can't find migration tags")
+
+    closest_version = get_version(closest_tag)
+    if not closest_version:
+        raise Exception(f"Can't determine version from {closest_tag}")
+
+    candidates = []
+
+    for migration_tag in migration_tags:
+        migration_version = get_version(
+            migration_tag.removeprefix(MIGRATION_TAG_PREFIX).removeprefix("v")
+        )
+
+        if not migration_version:
+            raise Exception(f"Can't determine migration version from {migration_tag}")
+
+        if closest_version == migration_version:
+            return migration_tag, True
+
+        if migration_version < closest_version:
+            candidates.append(migration_version)
+
+    if candidates:
+        closest_migration_version = sorted(candidates, reverse=True)[0]
+        return MIGRATION_TAG_PREFIX + "v" + str(closest_migration_version), False
+
+    raise Exception(f"Can't find matching migration version for {closest_tag}")

tests-e2e/test-migrate-from-legacy-infra

@@ -0,0 +1,31 @@
+#!/usr/bin/env bash
+
+set -xeuo pipefail
+
+#######################
+# Setup
+#######################
+
+source "$(realpath "$(dirname "${BASH_SOURCE[0]}")")/common.sh"
+
+init_project_dir
+
+# checkout a project before it was migrated to test against
+pushd "${PROJECT_DIR}"
+git remote add origin https://github.com/navapbc/platform-test
+# just before the Copier files were added in
+#
+# https://github.com/navapbc/platform-test/commit/7807c81d67328b9c23eebf73d2c6d0801795e2d0
+git fetch origin --depth=1 bb2e4a7b064c4f78eb8008e02d8f984a9b588209
+git checkout FETCH_HEAD
+popd
+
+expected_migration_point=platform-cli-migration/v0.12.4
+
+#######################
+# Test
+#######################
+
+$CMD infra migrate-from-legacy --commit "${PROJECT_DIR}"
+
+grep "_commit: ${expected_migration_point}" "${PROJECT_DIR}"/.template-infra/*.yml

tests/conftest.py

@@ -108,6 +108,13 @@ def infra_template_no_tags(
 def infra_template(infra_template_no_tags: InfraTemplateWritable) -> InfraTemplateWritable:
     template = infra_template_no_tags
     template.git_project.tag("v0.0.0")
+
+    template.git_project.checkout("-b", "migration-tag")
+    (template.template_dir / "migration-test.txt").write_text("foo")
+    template.git_project.commit_all("Migration checkpoint")
+    template.git_project.tag("platform-cli-migration/v0.0.0")
+    template.git_project.checkout("main")
+
     return template