🐛 Fix pvc-migration hook runAsNonRoot failure on non-OpenShift clusters by clubanderson · Pull Request #22947 · kubestellar/console

clubanderson · 2026-08-30T15:21:24Z

Problem

Console Live Promote has failed every scheduled run since 2026-08-26. The pre-upgrade pvc-migration hook pod is refused at container start on the console-live cluster:

Warning  Failed  pod/kc-live-kubestellar-console-pvc-migration-h4ldl
  Error: container has runAsNonRoot and image will run as root (container: migrate-pvcs)

The failed hook then cascades into pre-upgrade hooks failed → helm rollback → PersistentVolumeClaim ... spec is immutable → UPGRADE FAILED (run 33311744159).

Why the previous attempts did not clear it

[scanner] fix: make console-live-promote pvc-migration hook robust and debuggable #22908 added workflow-side diagnostics and stale-Job cleanup. Valuable (it surfaced the real error) but it never touched the hook's securityContext.
🐛 Fix pvc-migration hook pod failing runAsNonRoot check (#22919) #22923 set runAsNonRoot: false + runAsUser: 0 on the container. That would have passed the kubelet check, but it was replaced almost immediately by 🐛 Fix pvc-migration hook rejected by OpenShift restricted-v2 SCC #22929 (for Workflow failure: Build and Deploy KC #22925) because OpenShift's restricted-v2 SCC forbids runAsUser: 0. 🐛 Fix pvc-migration hook rejected by OpenShift restricted-v2 SCC #22929 reverted to runAsNonRoot: true with no runAsUser — which works on OpenShift (the SCC injects an in-range UID) but reintroduced the failure on every non-OpenShift cluster: alpine/k8s has no USER directive, so with nothing injecting a UID the container's effective user is root and the kubelet refuses to start it. The console-live cluster (OKE, oci-bv storage) is plain Kubernetes, so every promote since has failed.

Fix (differs from both prior attempts)

Mirror the chart's long-standing pattern for the app container (securityContext.runAsUser: 1001 in values.yaml, nulled on OpenShift):

values.yaml: new migration.runAsUser: 65532 / migration.runAsGroup: 65532 — explicit numeric non-root UID (distroless "nonroot" convention; the UID need not exist in /etc/passwd, and the hook already sets HOME=/tmp so kubectl cache/config writes work for any UID).
templates/pre-upgrade-pvc-migration.yaml: render runAsUser/runAsGroup conditionally so nulling them drops the fields; runAsNonRoot: true is kept (PodSecurity restricted still satisfied).
values-openshift.yaml + the OpenShift deploy_helm in build-deploy.yml + README: null migration.runAsUser/runAsGroup so restricted-v2 keeps injecting its own in-range UID — OpenShift behavior is unchanged from today.

Verified with helm template: default values render runAsUser/runAsGroup: 65532 on the migrate-pvcs container; the OpenShift overlay renders neither field.

Fixes #22946
Refs #22907 #22919

…rs (#22946) The pre-upgrade pvc-migration hook has been failing every scheduled Console Live Promote run with: Error: container has runAsNonRoot and image will run as root (container: migrate-pvcs) alpine/k8s has no USER directive, so it runs as root. After #22929 removed the explicit runAsUser (to satisfy OpenShift's restricted-v2 SCC, #22925), the hook was left with runAsNonRoot: true and no numeric UID. On OpenShift the SCC injects an in-range UID, so it works there — but on the plain-Kubernetes console-live cluster (OKE) nothing injects a UID, the container's effective user stays root, and the kubelet refuses to start it. The failed hook then cascades into 'pre-upgrade hooks failed' and an immutable-PVC rollback failure. Fix: mirror the chart's existing securityContext.runAsUser pattern for the app container — default migration.runAsUser/runAsGroup to the numeric non-root UID 65532 in values.yaml (rendered conditionally in the hook template), and null them on OpenShift via values-openshift.yaml and the build-deploy.yml OpenShift helm invocation, where the SCC injects its own UID. The UID need not exist in /etc/passwd, and the hook already sets HOME=/tmp so kubectl cache writes work for any UID. Verified with helm template: default values render runAsUser/runAsGroup 65532; the OpenShift overlay renders neither field. Fixes #22946 Refs #22907 #22919 Signed-off-by: Andrew Anderson <andy@clubanderson.com>

netlify · 2026-08-30T15:21:29Z

✅ Deploy Preview for kubestellarconsole canceled.

Name	Link
🔨 Latest commit	`d1fca95`
🔍 Latest deploy log	https://app.netlify.com/projects/kubestellarconsole/deploys/6a944a76031f9c00089c8d8f

kubestellar-prow · 2026-08-30T15:21:29Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign eeshaansa for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

github-actions · 2026-08-30T15:21:32Z

👋 Hey @clubanderson — thanks for opening this PR!

🤖 This project is developed exclusively using AI coding assistants.

Please do not attempt to code anything for this project manually.
All contributions should be authored using an AI coding tool such as:

Claude Code (Opus 4.5 / 4.6) — recommended

GitHub Copilot

Cursor

Other AI coding assistants

This ensures consistency in code style, architecture patterns, test coverage,
and commit quality across the entire codebase.

This is an automated message.

github-actions · 2026-08-30T15:21:34Z

🐝 Hi @clubanderson! I'm kubestellar-hive[bot], an automation bot for this repo.

Trusted users — org members and contributors with write access — can mention @kubestellar-hive in a comment to trigger repo automation.
On issues, that mention queues an automated fix attempt. On pull requests, it records extra context for existing automation.
This is not an interactive Q&A bot, so mentions should be treated as requests for automation rather than a conversation.

Automation may take a moment to start, and follow-up happens through workflow activity rather than chat replies.

Copilot

Pull request overview

This PR fixes the Helm chart’s pre-upgrade pvc-migration hook failing runAsNonRoot admission on non-OpenShift clusters by introducing explicit non-root runAsUser/runAsGroup defaults for the migration container, while keeping OpenShift behavior unchanged by nulling those fields in the OpenShift overlay and deploy workflow.

Changes:

Add migration.runAsUser / migration.runAsGroup defaults (65532) in the chart’s values.yaml for plain Kubernetes clusters.
Update the pvc-migration hook template to conditionally render runAsUser/runAsGroup, and update the OpenShift overlay to null them so SCC can inject an in-range UID.
Update OpenShift deploy workflow and Helm README instructions to also null the migration UID/GID fields.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
deploy/helm/kubestellar-console/values.yaml	Adds default non-root UID/GID for the pvc-migration hook container.
deploy/helm/kubestellar-console/values-openshift.yaml	Nulls migration UID/GID so OpenShift SCC can inject an allowed UID.
deploy/helm/kubestellar-console/templates/pre-upgrade-pvc-migration.yaml	Conditionally renders migration container `runAsUser`/`runAsGroup` alongside `runAsNonRoot: true`.
deploy/helm/kubestellar-console/README.md	Documents nulling migration UID/GID when deploying to OpenShift without the overlay file.
.github/workflows/build-deploy.yml	Ensures OpenShift deploy job also nulls migration UID/GID via Helm `--set`.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

            runAsNonRoot: true
+            {{- if .Values.migration.runAsUser }}
+            runAsUser: {{ .Values.migration.runAsUser }}
+            {{- end }}
+            {{- if .Values.migration.runAsGroup }}
+            runAsGroup: {{ .Values.migration.runAsGroup }}
+            {{- end }}
            allowPrivilegeEscalation: false


github-actions · 2026-08-30T16:27:48Z

Thank you for your contribution! Your PR has been merged.

Check out what's new:

KubeStellar Console — Live multi-cluster dashboard
Marketplace — Community extensions
Knowledge Base — Troubleshooting and how-tos

Stay connected: Slack #kubestellar-dev | Multi-Cluster Survey

github-actions · 2026-08-30T16:29:29Z

Post-merge build verification passed ✅

Both Go and frontend builds compiled successfully against merge commit 98f9f21fcc558c9f1f61ecf1e85aaf2b68f62f67.

…#22958) Console Live Promote failed every scheduled run with: serviceaccount "kc-live-kubestellar-console" not found (run 33346472599). The pvc-migration hook itself now completes fine (#22947); the missing SA is the *main* release ServiceAccount that the Deployment's ReplicaSet references. Root cause: both the canary and live releases deploy with serviceAccount.create: false, so no release owns the SA. But the corrupted release state left by the earlier hook-failure/rollback loop (#22946) still records the SA inside the perpetual rollback-target revision's manifest. Helm v4 therefore treats every upgrade as "SA removed from the chart" and prunes it mid-upgrade; pod creation is rejected instantly, the 10m --wait expires, and the atomic rollback then wedges on the immutable PVC volumeName patch. The follow-up 'helm rollback' restores the SA from the old revision, re-arming the same failure for the next scheduled run. Fix (workflow-only, no chart change needed): - live prod values now set serviceAccount.create: true (same name via LIVE_SERVICE_ACCOUNT), so the SA is part of every rendered upgrade and Helm reconciles/adopts it instead of pruning it. Once one upgrade succeeds, the release snapshot also re-owns the PVC, so future rollbacks no longer hit the immutability error. - new idempotent pre-flight step recreates the SA if absent before the canary/live deploys, unwedging the currently-broken cluster without any manual action. The canary release keeps create: false and only references the SA; exactly one release may own it. Fixes #22957 Refs #22946 Signed-off-by: Andy Anderson <andy@clubanderson.com>

Copilot AI lite review requested due to automatic review settings August 30, 2026 15:21

kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Aug 30, 2026

kubestellar-prow Bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Aug 30, 2026

github-actions Bot added the tier/3-restricted label Aug 30, 2026

Copilot started reviewing on behalf of clubanderson August 30, 2026 15:21

View session

Copilot AI reviewed Aug 30, 2026

View reviewed changes

hivecommons-hive Bot added agent/scanner Filed by the scanner agent hive/hosted-kubestellar-console-4vkt labels Aug 30, 2026

hivecommons-hive Bot merged commit 98f9f21 into main Aug 30, 2026
33 of 34 checks passed

kubestellar-prow Bot deleted the fix/live-promote-runasnonroot branch August 30, 2026 16:27

clubanderson mentioned this pull request Aug 31, 2026

🐛 fix: stop Helm pruning the console-live ServiceAccount during promote #22958

Merged

clubanderson mentioned this pull request Sep 3, 2026

🐛 fix: bootstrap live test user on kc-live so the signed-session promote gate can pass #23090

Merged

Navigation Menu

Repository navigation

🐛 Fix pvc-migration hook runAsNonRoot failure on non-OpenShift clusters - #22947

🐛 Fix pvc-migration hook runAsNonRoot failure on non-OpenShift clusters#22947
hivecommons-hive[bot] merged 1 commit into
mainfrom
fix/live-promote-runasnonroot

clubanderson commented Aug 30, 2026

Uh oh!

netlify Bot commented Aug 30, 2026 •

edited

Loading

Uh oh!

kubestellar-prow Bot commented Aug 30, 2026

Uh oh!

github-actions Bot commented Aug 30, 2026

Uh oh!

github-actions Bot commented Aug 30, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

github-actions Bot commented Aug 30, 2026

Uh oh!

github-actions Bot commented Aug 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

clubanderson commented Aug 30, 2026

Problem

Why the previous attempts did not clear it

Fix (differs from both prior attempts)

Uh oh!

netlify Bot commented Aug 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for kubestellarconsole canceled.

Uh oh!

kubestellar-prow Bot commented Aug 30, 2026

Uh oh!

github-actions Bot commented Aug 30, 2026

Uh oh!

github-actions Bot commented Aug 30, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

github-actions Bot commented Aug 30, 2026

Uh oh!

github-actions Bot commented Aug 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

netlify Bot commented Aug 30, 2026 •

edited

Loading