Skip to content

Commit 10ad4dc

Browse files
ci: bump kubectl rollout-status timeout 180s → 300s
Today's #198 deploy timed out at 180s even though the new ReplicaSet came up healthy (rollout actually took ~240s on a fresh node with a cold image pull). The job exited red, prod was correct, and the false-failure forced a manual rerun + on-call investigation. Bumping to 300s removes the rerun cycle. Same change ships simultaneously in worker + provisioner deploy.yml. Healthz verify loop extended 6×5s → 12×5s for the same reason: if rollout takes the full timeout, the new pods are only just live when verify starts and the public LB needs a few seconds to route. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 1057506 commit 10ad4dc

1 file changed

Lines changed: 9 additions & 3 deletions

File tree

.github/workflows/deploy.yml

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -278,10 +278,14 @@ jobs:
278278
"deployment/${K8S_DEPLOYMENT}" \
279279
"${K8S_CONTAINER}=${IMAGE}" \
280280
-n "${K8S_NAMESPACE}"
281+
# 2026-05-30: bumped 180s → 300s after a real rollout took ~240s
282+
# (image pull + readiness on a fresh node) and the workflow timed
283+
# out even though the pod came up healthy. The job exited red but
284+
# prod was correct, so the false-failure forced a manual rerun.
281285
kubectl rollout status \
282286
"deployment/${K8S_DEPLOYMENT}" \
283287
-n "${K8S_NAMESPACE}" \
284-
--timeout=180s
288+
--timeout=300s
285289
286290
- name: Verify rolled-out image tag matches built version
287291
run: |
@@ -298,8 +302,10 @@ jobs:
298302
- name: Curl live /healthz and confirm new SHA is reported
299303
run: |
300304
SHORT_SHA="${{ steps.meta.outputs.short_sha }}"
301-
# Allow up to ~30s for the new pod to start serving the public URL.
302-
for i in 1 2 3 4 5 6; do
305+
# 2026-05-30: extended 30s → 60s to match the rollout-timeout bump.
306+
# If rollout took the full 5 min, the new pods are only just live
307+
# when this step starts and the public LB needs a few s to route.
308+
for i in $(seq 1 12); do
303309
BODY=$(curl -fsSL --max-time 5 "${HEALTHZ_URL}" || echo "")
304310
echo "Attempt ${i}: ${BODY}"
305311
if echo "${BODY}" | grep -q "${SHORT_SHA}"; then

0 commit comments

Comments
 (0)