Repository navigation
feat: make reboot — diagnose + auto-heal the lab after a host reboot - #17
Merged
Merged
Conversation
A host reboot restarts the k3d node container and breaks several things that a
plain `kubectl get pods` won't explain, and they compound:
- flannel /run/flannel/subnet.env wiped (tmpfs) -> pod sandbox failures
- host.k3d.internal stripped from CoreDNS -> ESO can't reach floci
- webapp image drops out of node containerd -> ImagePullBackOff (node can't pull
the floci ECR ref: inside the node it resolves to the node's own loopback)
- floci down / lost un-flushed data
New:
- scripts/recover.sh (make reboot): checks and fixes each of the above in order,
idempotent, never touches Terraform state / floci secrets / infra roots.
- scripts/ensure-webapp-image.sh: snapshotter-safe image loader. `k3d image
import` silently no-ops on Docker's containerd-snapshotter store (default on
recent Docker/Ubuntu) — docker save emits an OCI layout the importer rejects
("content digest not found") but reports success. This verifies the image
landed and falls back to pulling the identical upstream image into the node's
containerd + retagging. Used by both `make reboot` and the deploy.
- terraform/cluster/webapp.tf: image import now calls ensure-webapp-image.sh, so
a fresh `make deploy` works regardless of Docker's storage driver.
- README: "After a reboot" section + the image-loading note.
Validated end-to-end on a live cluster: healthy checks pass, host-fix reapplied,
ESO re-synced (SecretSynced), webapp 1/1, app HTTP 200. Security gate PASS.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A fresh Ubuntu reproducibility test surfaced that a host reboot leaves the lab broken in ways
kubectl get podsdoesn't explain, and the failures compound:failed to setup network ... /run/flannel/subnet.envSecretSyncedError,lookup host.k3d.internal ... i/o timeouthost.k3d.internalfrom CoreDNS on restartImagePullBackOff,dial tcp 127.0.0.1:5100: connection refusedPlus a distinct root cause for the image never landing on modern Docker: the containerd-snapshotter image store (default on recent Docker/Ubuntu) makes
k3d image importsilently no-op —docker saveemits an OCI layout the importer rejects (content digest not found) while reporting success.What
make reboot(scripts/recover.sh) — diagnoses and auto-heals, in order: floci up → cluster started → flannel (bounces node ifsubnet.envmissing) →host.k3d.internal→ webapp image → ESO re-sync → log shipper → webapp rollout → status + app HTTP code. Idempotent; never touches Terraform state, floci secrets, or infra/secrets roots.scripts/ensure-webapp-image.sh— snapshotter-safe image loader: verifies the image actually landed on the node and falls back to pulling the identical upstream image into the node's containerd + retagging. Used by bothmake rebootand the deploy.terraform/cluster/webapp.tf— image import now calls the loader, so a freshmake deployworks regardless of Docker's storage driver.Verified
Ran
make rebooton a live cluster: healthy checks pass, host-fix reapplied, ESOSecretSynced, webapp1/1 Running, app HTTP 200. Security gate (kubesec/tfsec/conftest) PASS.🤖 Generated with Claude Code