Repository navigation
🐛 [scanner] fix: default Deployment strategy to Recreate to stop vllm-d deploy deadlock - #23578
Conversation
…-d deploy deadlock
The kc Deployment is single-replica (replicaCount: 1) and backed by a
ReadWriteOnce/block-storage PVC on the vllm-d cluster
(ibmc-vpc-block-10iops-tier). values.yaml already documented that
Recreate is required in this case to avoid the RollingUpdate deadlock,
but the default strategy.type was still RollingUpdate.
With RollingUpdate, the new pod comes up while the old pod is still
terminating and holding the volume, so the CSI driver rejects the
concurrent mount with 'an operation with the given Volume ID ...
already exists', the Deployment gets stuck 'Pending termination', and
helm upgrade eventually times out ('context deadline exceeded'). This
caused run 35408410648 of deploy-vllm-d to fail after both retry
attempts (issue #23574).
Switching the default to Recreate makes Kubernetes fully terminate the
old pod (and release the volume) before starting the new one, matching
the documented intent and the vllm-d job's single-replica/RWO setup.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: GitHub Copilot <223556219+Copilot@users.noreply.github.com>
kubestellar-prow
Bot
added
the
dco-signoff: yes
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
✅ Deploy Preview for kubestellarconsole canceled.
|
|
👋 Hey @kubestellar-hive[bot] — thanks for opening this PR!
This is an automated message. |
kubestellar-prow
Bot
added
the
size/S
|
Thank you for your contribution! Your PR has been merged. Check out what's new:
Stay connected: Slack #kubestellar-dev | Multi-Cluster Survey |
|
Post-merge build verification passed ✅ Both Go and frontend builds compiled successfully against merge commit |
Fixes #23574
Root cause
Run 35408410648 of
deploy-vllm-dfailed on both retry attempts with:with repeated events:
The
kcDeployment is single-replica (replicaCount: 1) and its PVC is backed by block storage on the vllm-d cluster (ibmc-vpc-block-10iops-tier).values.yamlalready documented thatRecreateis required for single-replica deployments on this kind of storage to avoid a RollingUpdate deadlock, but the chart's defaultdeployment.strategy.typewas stillRollingUpdate. With RollingUpdate, the new pod starts while the old pod is still terminating and holding the volume, so the CSI driver rejects the concurrent mount, the Deployment gets stuck "Pending termination", andhelm upgradeeventually times out.Fix
Changed the chart's default
deployment.strategy.typefromRollingUpdatetoRecreateindeploy/helm/kubestellar-console/values.yaml, matching the documented intent and the vllm-d job's single-replica/RWO setup. No workflow file or job overrides this value, so the fix takes effect on the next deploy without further changes.This is a config-only change; not a code refactor.
— hive: agent=scanner backend=copilot model=claude-fable-5 copilot=1.0.78