Skip to content

Main branch broken: Build and Deploy KC #24244

Closed
Closed
Main branch broken: Build and Deploy KC#24244
Labels
help wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.hive/likely-doneHive verified that a merged PR references or claims this issue; pending confirmationmain-brokenAuto-applied to last merged PR when main branch CI fails

Description

@github-actions
github-actions
bot
opened on Oct 11, 2026

🚨 Main branch is red

Build Sheriff on call: @clubanderson @eeshaanSA @MikeSpreitzer @Nupurshivani — the 4-hour recovery SLA clock started at 2026-10-11 22:49 UTC (when the gate finished failing).

Detail Value
Workflow Build and Deploy KC (.github/workflows/build-deploy.yml)
Run #38180179764
Commit 71d9147
Source PR #24241 (labelled main-broken)
Failed jobs deploy-vllm-d

Playbook

Follow the Main Branch Build Recovery SLA in docs/INCIDENT-RESPONSE.md:

  1. Triage (0-30 min): identify the failing commit and decide flaky vs. real breakage
  2. Resolve (30 min - 4 h): revert (preferred) or forward-fix; escalate after 2 h
  3. Verify: wait for main to go green, then close this issue and remove the main-broken label
  4. Post-mortem (within 24 h): document root cause and prevention in this issue

Opened automatically by the Main Branch Broken workflow. Repeat failures are appended as comments.

Activity

  1. github-actions
    added
    main-brokenAuto-applied to last merged PR when main branch CI fails
    on Oct 11, 2026

  2. hivecommons-hive commented on Oct 11, 2026

    @hivecommons-hive
    hivecommons-hiveBot
    on Oct 11, 2026 – with hivecommons-hive
    Contributor

    Triage (scanner): run 38180179764 deploy-vllm-d failed at "Deploy with Helm" — the new pod kc-kubestellar-console-66b5bf45fc-fz8db is stuck Init:CreateContainerError until the Helm wait times out.

    Root cause (from job diagnostics): SELinux relabel of the console data PVC fails —

    Error: relabel failed .../pvc-69c4a202-733e-4c95-9ff7-6a8bb396d8a1/mount:
lsetxattr .../mount/console-20260805-000001.db: permission denied

    This is not caused by commit 71d9147 — that commit (PR #24241, fsGroupChangePolicy: OnRootMismatch) addresses fsGroup chown, but this failure is kubelet's recursive SELinux relabel (lsetxattr), which fsGroupChangePolicy does not affect. The kubelet cannot set security.selinux xattrs on the existing console-*.db/.db-journal backup files on PV pvc-69c4a202-....

    This needs cluster-side action on the fmaas vllm-d cluster (not a repo code fix):

    • run restorecon -R / fix ownership on the volume's files so lsetxattr succeeds, or
    • if the CSI backend doesn't support SELinux xattrs (e.g. NFS-backed), mount with a context= mount option at the StorageClass/PV level so kubelet skips per-file relabel, or
    • recreate the PVC (data loss limited to local db backups per the restore flow).

    Flaky-vs-real verdict: real, infra/storage — every retry will hit the same relabel error until the volume is fixed. Deferring to the Build Sheriffs; no revert candidate exists since the breakage is independent of the source commit.

    🐝 Hive Agent: scanner | Instance: hosted-kubestellar-console-4vkt | SHA: 4195ab9

    — hive: agent=scanner backend=copilot model=claude-fable-5 copilot=1.0.88

  3. github-actions commented on Oct 12, 2026

    @github-actions
    github-actionsBot
    on Oct 12, 2026 – with GitHub Actions
    ContributorAuthor

    Still red — Build and Deploy KC failed again on main.

  4. github-actions commented on Oct 12, 2026

    @github-actions
    github-actionsBot
    on Oct 12, 2026 – with GitHub Actions
    ContributorAuthor

    Still red — Build and Deploy KC failed again on main.

  5. github-actions commented on Oct 12, 2026

    @github-actions
    github-actionsBot
    on Oct 12, 2026 – with GitHub Actions
    ContributorAuthor

    Still red — Build and Deploy KC failed again on main.

  6. github-actions commented on Oct 12, 2026

    @github-actions
    github-actionsBot
    on Oct 12, 2026 – with GitHub Actions
    ContributorAuthor

    Still red — Build and Deploy KC failed again on main.

  7. github-actions commented on Oct 12, 2026

    @github-actions
    github-actionsBot
    on Oct 12, 2026 – with GitHub Actions
    ContributorAuthor

    Still red — Build and Deploy KC failed again on main.

  8. hivecommons-hive commented on Oct 12, 2026

    @hivecommons-hive
    hivecommons-hiveBot
    on Oct 12, 2026 – with hivecommons-hive
    Contributor

    Update (scanner, 2026-10-12): two distinct failure causes now on main:

    1. Persistent — deploy-vllm-d: kubelet SELinux relabel of the console data PVC fails (lsetxattr ... permission denied on existing console-*.db backups). Cluster-side storage fix still required (see earlier diagnosis).
    2. Transient (new, run 38245463581) — build (linux/amd64) step "Build and push by digest" failed with public.ecr.aws/docker/library/alpine:3.24 ... 429 Too Many Requests — registry rate-limit, not a repo regression. Will clear on re-run; if it recurs, consider mirroring the base image to ghcr.io or adding registry auth.

    No repo-side code change needed for either; leaving open until deploy-vllm-d passes.

    🐝 Hive Agent: scanner | Instance: hosted-kubestellar-console-4vkt | SHA: befb46d

    — hive: agent=scanner backend=copilot model=claude-fable-5 copilot=1.0.88

  9. hivecommons-hive commented on Oct 12, 2026

    @hivecommons-hive
    hivecommons-hiveBot
    on Oct 12, 2026 – with hivecommons-hive
    Contributor

    Triage update (scanner): newest failure run 38245463581 (2026-10-12 11:58 UTC) is a different, transient cause — build (linux/amd64) step "Build and push by digest" failed pulling the base image:

    public.ecr.aws/docker/library/alpine:3.24@sha256:294b683c...: 429 Too Many Requests

    That is a registry rate-limit flake, not code; a re-run should pass the build job. The persistent deploy-vllm-d failure (SELinux lsetxattr ... permission denied on PV pvc-69c4a202-...) still requires cluster-side storage action as diagnosed above. No repo code fix applicable for either.

    🐝 Hive Agent: scanner | Instance: hosted-kubestellar-console-4vkt | SHA: 3c4a38f

    — hive: agent=scanner backend=copilot model=claude-fable-5 copilot=1.0.88

  10. hivecommons-hive commented on Oct 12, 2026

    @hivecommons-hive
    hivecommons-hiveBot
    on Oct 12, 2026 – with hivecommons-hive
    Contributor

    Triage update (scanner, 2026-10-12): latest failure run 38245463581 failed at a DIFFERENT step than the ongoing incident — the build (linux/amd64) job hit a transient registry rate-limit pulling the base image:

    public.ecr.aws/docker/library/alpine:3.24 ... 429 Too Many Requests

    That is a one-off infra blip, not the SELinux PVC relabel issue tracked above (which still blocks deploy-vllm-d on every run that reaches it). No repo code fix applies to either failure mode; the PVC relabel still needs cluster-side action on the fmaas vllm-d cluster.

    🐝 Hive Agent: scanner | Instance: hosted-kubestellar-console-4vkt | SHA: 725488c

    — hive: agent=scanner backend=copilot model=claude-fable-5 copilot=1.0.88

  11. github-actions commented on Oct 12, 2026

    @github-actions
    github-actionsBot
    on Oct 12, 2026 – with GitHub Actions
    ContributorAuthor

    Still red — Build and Deploy KC failed again on main.

  12. github-actions commented on Oct 12, 2026

    @github-actions
    github-actionsBot
    on Oct 12, 2026 – with GitHub Actions
    ContributorAuthor

    Still red — Build and Deploy KC failed again on main.

  13. github-actions commented on Oct 12, 2026

    @github-actions
    github-actionsBot
    on Oct 12, 2026 – with GitHub Actions
    ContributorAuthor

    Still red — Build and Deploy KC failed again on main.

  14. hivecommons-hive
    added
    🐝 covered by PRHive verified that an open PR references or claims this issue; still actionable until confirmed
    on Oct 13, 2026
  15. github-actions
    added
    help wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.
    on Oct 13, 2026
  16. hivecommons-hive
    added
    hive/likely-doneHive verified that a merged PR references or claims this issue; pending confirmation
    and removed
    🐝 covered by PRHive verified that an open PR references or claims this issue; still actionable until confirmed
    on Oct 13, 2026

  17. hivecommons-hive commented on Oct 13, 2026

    @hivecommons-hive
    hivecommons-hiveBot
    on Oct 13, 2026 – with hivecommons-hive
    Contributor

    Fixed by #24268 (merged 147eebe). Closing. Reply /reopen if the problem persists.

  18. hivecommons-hive
    closed this as completedon Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    help wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.hive/likely-doneHive verified that a merged PR references or claims this issue; pending confirmationmain-brokenAuto-applied to last merged PR when main branch CI fails

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions