vault-k8s goes into "Please unseal Vault" blocked state during test_model_controller_migration, causing JujuWaitTimeoutError

[motjuste]

🤖 This issue was drafted with AI assistance (GitHub Copilot / Test Observer MCP) from Test Observer data, Juju logs, and repo source. @canonical/solutions-qa please review before triage.

Environment

• vault-k8s rev 572 (2.0/beta)
• grafana-k8s rev 200 (12.4/stable)
• juju 3.6.28
• Base: juju:3/stable ubuntu:24.04 kubernetes

Test Observer link
https://test-observer.canonical.com/#/charms/410524?testExecutionId=734881&testResultId=13829728

Error

juju.backend.JujuWaitTimeoutError: Timed out while waiting for applications: [active], units: [active], unit agents: [idle] (applications: ['target'], units: ['target/0'])

Test log excerpt (captured log, test_model_controller_migration)

INFO Migrating model '<redacted-model>' from source controller '<redacted-controller>' to target controller '<redacted-controller>'
INFO Waiting for vault charm 'vault-k8s' application 'target' to be scaled
INFO Waiting for vault charm 'vault-k8s' application 'target' units to be settled
INFO Vault charm 'vault-k8s' unit 'target/leader' is already initialized, skipping init/unseal
INFO Skipping authorizing vault charm 'vault-k8s' unit 'target/leader'
INFO Waiting 0:15:00 for model ... to exist before continuing.
INFO Waiting 0:15:00 to be idle.
INFO Restarting model controller.
INFO Rollout restart initiated for Statefulset: 'controller' in namespace 'controller-<redacted>'
INFO Successfully rolled out generation '3'.
INFO Waiting 0:15:00 to be idle.
ERROR Failure in test_model_controller_migration: juju.backend.JujuWaitTimeoutError: Timed out while waiting for applications: [active], units: [active], unit agents: [idle] (applications: ['target'], units: ['target/0'])

Juju status at failure (17:03:28Z)

App       Version  Status   Scale  Charm        Channel      Rev  Message
neighbor  12.4.2   active       1  grafana-k8s  12.4/stable  200
target             blocked      1  vault-k8s    2.0/beta     572  Please unseal Vault

Unit         Workload  Agent  Message
neighbor/0*  active    idle
target/0*    blocked   idle   Please unseal Vault

(Note: target/0 was active/idle at the start of this same test run, per the initial "Juju status" snapshot logged at 16:32:30Z — Vault was fine before this migration cycle began.)

Observed timeline (from debug-log.txt / status-log/unit-target-0.yaml, timestamps UTC)

• 16:44:05 — second migrate_model() call begins (migration phase → IMPORT).
• 16:46:21 / 16:46:24 — GET /v1/sys/seal-status HTTP/1.1" 200 325 (Vault reports unsealed at this point).
• 16:48:34–16:51:21 — target/0 runs its post-migration hook sequence (start → stop → pebble-ready → ... per uniter operation log).
• 17:05:49.643Z — status-log records:

  - status: blocked
    message: Please unseal Vault
    since: 2026-09-15T17:05:49.643025781Z
    type: workload
  

• Test subsequently times out waiting for target/target/0 to reach active/idle.

Code context (for maintainers)
UnsealVaultK8sJujuExtension.post_migrate_model (charm_integration_testing/extensions/unseal_vault/extensions.py) already calls try_init_or_unseal_all_vaults(..., authorize_charm=False) (vault_unsealer.py) right after migrate_model() returns — this is the one-shot check that produced the seal-status calls at 16:46:21/16:46:24 above. This confirms the extension already anticipates Vault may need re-unsealing after a migration.

However, that check runs before Juju's own post-migration unit hook lifecycle (stop → upgrade-charm → config-changed → start → vault-pebble-ready, observed 16:48:34–16:51:21), which restarts the vault workload container. No later hook in extensions.py re-verifies seal status after this lifecycle completes, before test_model_controller_migration.py proceeds to wait for active/idle.

Steps to reproduce

1. Deploy vault-k8s (and grafana-k8s) per the standard bundle.
2. Run test_model_controller_migration from charm-integration-testing.
3. Observe target/0 become blocked with "Please unseal Vault" after the second migration.

Related
A search of Test Observer history (2026-04-03 to 2026-09-01) for the same vault-k8s:status = blocked:Please unseal Vault signature during test_model_controller_migration found 168 matching failed test results across 14 charms — 135 with vault-k8s as the charm under test, and 33 across 13 other charms (postgresql-k8s, opentelemetry-collector-k8s, traefik-k8s, mongodb-k8s, kafka-k8s, pgbouncer-k8s, prometheus-k8s, mysql-router-k8s, tempo-coordinator-k8s, zookeeper-k8s, temporal-k8s, sloth-k8s, kratos) that deploy vault-k8s as a supporting/fixture charm. No prior GitHub issue was found tracking this specific symptom.

---

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

[syncronize-issues-to-jira]

Thank you for reporting your feedback to us!

The internal ticket has been created: https://warthogs.atlassian.net/browse/SQT-1263.

This message was autogenerated