up hangs indefinitely in "Waiting for services to be healthy" when the log tail dies (expired credentials, both AWS and GCP)

[defangdevs]

Summary

defang up can hang indefinitely in "Waiting for services to be healthy..." after a fully successful deployment, when the log-tail stream dies in a way the tailer cannot recover from. The deployment status wait never completes even though the services are healthy, so the command only ends when something external kills it.

Observed on both AWS and GCP in the same CI run, with CLI 3.13.1-d8aa2a0a-nightly.

Evidence: defang-mvp run 32537540857 (new-provider-sanity.yml, 30-minute job timeout).

AWS

Timeline:

time (UTC)	event
23:39:25	* Deployment complete. Waiting for services to be healthy...
23:41:26	both ECS services reach steady state (app, smokeworker — ACTIVE, 1/1 running, rolloutState: COMPLETED)
23:39:25 → 00:07:54	CLI keeps polling, prints nothing
00:07:54	job timeout kills it; CLI then surfaces the error below

AccessDeniedException: The security token included in the request is expired
  — operation error CloudWatch Logs: StartLiveTail
Tail while monitoring stopped with ... AccessDeniedException: The security token included in the request is expired

So the CLI sat for 26 minutes after the infrastructure was already healthy. I confirmed the ECS state independently via the AWS API, and the ALB returned HTTP 200 with the expected content the whole time.

Root cause looks like: the CI's assumed-role credentials expired partway through the long-lived tail stream, and the tail/monitor loop kept retrying against dead credentials instead of refreshing them or failing fast. Because the status wait is coupled to that stream, it never observed the healthy state.

GCP

Same shape, different transport — a reconnect loop that goes quiet rather than erroring:

- Disconnected: rpc error: code = Unavailable desc = ... read: connection reset by peer
! Reconnecting...
- Closing log tailer / Closing log tailer client
- Closing log lister client
   <12 minutes of complete silence, then killed>

Deployment had completed at 23:44:58 and the Compute Engine instance was RUNNING with its container serving (verified directly in GCP).

Why this matters

• It is indefinite, not slow. Raising CI timeouts does not help; a 30-minute budget was already ~26 minutes more than the deploy needed.
• It fails a green deployment. Both providers deployed correctly; only the CLI's observation of that fact broke. In CI this reads as a red run for a working change.
• Neither leg ever printed DEPLOYMENT_COMPLETED, so there is no signal that the wait is stuck rather than progressing.

Suggestions

1. Refresh credentials on the tail path rather than reusing a credential set captured at stream start — an expired token should be a refresh, not a terminal-but-silent retry loop.
2. Decouple the deployment status wait from log tailing, so a dead tail degrades log output without blocking up from observing steady state.
3. Fail fast and loudly on a non-retryable tail error (AccessDeniedException is not worth retrying), and/or fall back to polling.
4. Emit periodic progress while waiting, so a stuck wait is distinguishable from a slow one.

Point 2 and 3 overlap with #2071, which reports the same tailer fragility via a different trigger (LimitExceededException: Rate exceeded) and already suggests falling back to polling when live tail fails. This issue adds the expired-credential trigger and the more severe symptom — the status wait hanging indefinitely on healthy infrastructure.