Repository navigation
ResourceQuota admission rejections fail tasks instead of waiting for capacity #8233
Description
Description
When Kubernetes rejects Pod creation because a namespace ResourceQuota is temporarily exhausted, Flyte treats the rejection as a user-retryable task failure.
Tasks without user retries fail immediately. With retries enabled, repeated admission rejections consume attempts even though the task never starts.
Environment
- Flyte backend: v2.0.52
- Flyte Python SDK: 2.12.0
- Kubernetes with namespace ResourceQuota enabled
Steps to reproduce
- Configure a namespace memory quota.
- Run tasks concurrently so their combined resource requirements exceed the available quota, while each individual task fits within the hard limit.
- Configure no user retries.
- Observe tasks whose Pod creation is rejected by quota admission.
We reproduced this with two overlapping workflows, each launching ten parallel tasks requesting 1Gi memory with a 2Gi limit, under a 32Gi memory-limit quota.
Actual behavior
Quota-rejected tasks fail with a user-retryable RuntimeFailure, or fail immediately when no retries are configured.
Example Kubernetes rejection:
exceeded quota: tasks, requested: limits.memory=2Gi, used: limits.memory=32640Mi, limited: limits.memory=32Gi
No Pod exists for the rejected attempt, so waiting on Pod scheduling cannot resolve this.
Expected behavior
- Temporary quota contention should produce WAITING_FOR_RESOURCES and retry admission on the same attempt.
- Waiting should consume neither user nor system retries and should not start the execution-timeout clock.
- Waiting should have a configurable timeout and survive executor restarts.
- A request exceeding the hard quota should fail permanently rather than wait for capacity that cannot accommodate it.
- Other Forbidden responses, such as RBAC and webhook denials, should retain their existing handling.
Impact
Normal parallel workloads can fail solely because other tasks temporarily occupy quota. Increasing retries only delays the problem and consumes task attempts.
Validation
With a patched Executor, both overlapping workflows completed successfully: all 22 actions succeeded on attempt one, including a quota-blocked child that later gained admission.
A separate 17Gi memory request against a 16Gi request quota failed permanently without creating a Pod.
This concerns quota admission, not physical node capacity or FIFO scheduling.
Activity
runllm commentedon Oct 23, 2026
It could be a real gap, but the provided sources don’t document Kubernetes
ResourceQuotaPod-admission rejections or whether Flyte can retry admission on the same attempt. They do distinguish task-code retries from platform-handled infrastructure retries, but don’t say how quota errors are classified (Retries and timeouts).Would you like me to assess the issue’s technical framing, suggest an implementation/acceptance criteria, or rewrite it for a GitHub issue?
Sources:
This is an AI-generated response and your feedback is appreciated! Please leave a 👍 if this is helpful and 👎 if it is not.
Mention @Herald to ask follow-up questions. See runllm.com for more!
kraiper commentedon Oct 23, 2026
I potential solution is implemented here #8209
kraiper commentedon Oct 23, 2026
Thanks for the reference. This issue occurs before task execution: Kubernetes rejects Pod creation for quota exhaustion, and the executor classifies that as a user-retryable failure. Retry/backoff settings therefore consume attempts rather than wait on the same attempt. The documented quota-waiting lifecycle matches the behavior this fix implements. We should consider how the admission wait integrates with max_queued_time.
runllm commentedon Oct 23, 2026
PR #8209 proposes the requested behavior: temporary quota
Forbiddenresponses wait inWAITING_FOR_RESOURCESand retry admission on the same attempt; requests exceeding hard quota fail permanently, and continued rejections can time out. It also preserves otherForbiddenhandling and persists the wait start across reconciliations and executor restarts.The supplied source does not specify how this quota wait interacts with
max_queued_time, so that remains an open integration question. Would you like more detail?Sources:
This is an AI-generated response and your feedback is appreciated! Please leave a 👍 if this is helpful and 👎 if it is not.
Mention @Herald to ask follow-up questions. See runllm.com for more!
kraiper commentedon Oct 23, 2026
Should max_queued_time also bound this pre-Pod admission wait? If so, how should it interact with the proposed quota-wait-timeout setting?
runllm commentedon Oct 23, 2026
Yes—if quota admission is part of Waiting for resources,
max_queued_timeshould bound the total pre-execution wait, including the time before a Pod exists. Flyte documentsmax_queued_timeas covering Queued and Waiting for resources, per attempt; it does not specify how it interacts with this proposed admission wait (Retries and timeouts).Treat the quota-wait timeout as a quota-specific cap, not extra time: whichever deadline comes first should end the wait. Keep the PR’s hard-quota permanent failure, and preserve existing handling for unrelated
Forbiddenerrors (PR #8209). The supplied sources don’t define whether both settings should be configurable independently or which error should be reported when the limits coincide. Would you like more detail?Sources:
This is an AI-generated response and your feedback is appreciated! Please leave a 👍 if this is helpful and 👎 if it is not.
Mention @Herald to ask follow-up questions. See runllm.com for more!
kraiper commentedon Oct 23, 2026
@Herald
That policy is reasonable, but the timeout outcome matters too, not just which timer wins:
max_queued_time should retain its original per-attempt start, including time before Pod creation.
quota-wait-timeout starts at the first quota rejection; it grants no additional queue time.
The earlier deadline applies. On an exact tie, I’d prefer the standard max_queued_time outcome.
max_queued_time expiry retains its documented retry behavior; the PR currently makes QuotaWaitTimeout permanent. Those are different semantics.
Hard-quota violations remain immediate permanent failures; unrelated Forbidden handling stays unchanged.
Keeping both settings is defensible: one is a task-level queue budget, the other an operator-level quota-admission cap. These are proposed integration rules, not behavior already implemented in our patch.
runllm commentedon Oct 23, 2026
That’s a clear integration policy. It distinguishes the task’s per-attempt queue budget from a quota-specific cap: start
max_queued_timeat the attempt’s original queue time, start the quota timer at its first quota rejection, and enforce whichever expires first. On a tie, report the standardmax_queued_timeoutcome; preserve its retry behavior, whileQuotaWaitTimeoutremains permanent. Hard-quota violations and unrelatedForbiddenresponses keep their separate handling.These are proposed integration rules, not behavior specified by the sources: the timeout documentation defines
max_queued_timeas a per-attempt bound for queued and resource-waiting phases, while PR #8209 describes quota-specific waiting and timeout behavior without defining how the timers interact.Sources:
This is an AI-generated response and your feedback is appreciated! Please leave a 👍 if this is helpful and 👎 if it is not.
Mention @Herald to ask follow-up questions. See runllm.com for more!