fix(controller): bound CRD watch timeout to stop reflector staleness on AKS (#451)#456
Merged
Merged
Conversation
…on AKS (#451) Fixes #451 — on AKS, CR spec edits (InferencePolicy/ToolPolicy/EgressApproval/ …) intermittently stopped reaching the compiled ConfigMap until a controller restart. Root cause: every reconciler used `watcher::Config::default()`. That is NOT "no timeout" — kube-rs sends `timeoutSeconds=290` by default. The problem is the value: the Azure Standard LB idle timeout is 240s, so a quiet watch is silently dropped (no FIN/RST) at 240s, BEFORE the server's 290s close. kube-rs 3.1.0 then polls the dead stream with a bare `stream.next().await` (no client-side idle deadline) and hangs forever — the reflector store freezes while the 15s requeue keeps reconciling stale cached objects. Fix: a shared `watch_config::bounded()` that sets the watch timeout to 200s (< the 240s LB idle; kube-rs also rejects >=295s), applied to every primary and secondary (.owns/.watches) watch across all 11 CRD reconcilers so it can't diverge again. The server now closes each watch at 200s — that close is traffic that prevents the connection from ever sitting idle for 240s, and the watcher re-List/re-watches (refreshing the store) within a bounded window. Overridable via KARS_WATCH_TIMEOUT_SECS (clamped 30..=600) for other idle timeouts. This is the version-independent, low-risk prevention fix and works on the current kube-rs 3.1.0. Researched alternatives (documented in the module): - kube-rs 4.0 adds a client-side `next_with_idle_timeout` backstop (caps the freeze at ~295s instead of forever) but still ships NO HTTP/2 keepalive ping. - The architecturally superior, client-go-parity fix is HTTP/2 keepalive ping on the kube client (~45s dead-connection detection); it needs a custom hyper client stack + deps and is tracked as a defense-in-depth follow-up. Verification: clippy -D warnings clean, cargo fmt clean, 849 controller tests pass + 3 new watch_config unit tests (invariant: default < 240s; bounded() sets the timeout; env override clamped). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Scanned FilesNone |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #451.
Problem
On AKS, CR spec edits (
InferencePolicy/ToolPolicy/EgressApproval/ …) intermittently stopped reaching the compiled ConfigMap until a controller restart. The reconciler kept recompiling the pre-edit object on its 15s requeue (frozenversion_hash/compiled_digest) despite freshmetadata.generation— i.e. reconciling fresh metadata against a stale reflector store.Root cause (corrected & verified)
Every reconciler used
watcher::Config::default(). That is not "no timeout" — kube-rs sendstimeoutSeconds=290by default. The problem is the value:stream.next().await(no client-side idle deadline) → hangs forever → reflector store freezes.Fix
A shared
controller/src/watch_config.rs::bounded()sets the watch timeout to 200s (< the 240s LB idle; kube-rs rejects ≥295s), applied to every primary and secondary (.owns/.watches) watch across all 11 CRD reconcilers so it can't diverge again. The server now closes each watch at 200s — that close is traffic that prevents the connection from ever sitting idle for 240s, and the watcher re-List/re-watches (refreshing the store) within a bounded window. Overridable viaKARS_WATCH_TIMEOUT_SECS(clamped 30..=600).This is the version-independent, low-risk prevention fix on the current kube-rs 3.1.0.
Researched alternatives (documented in the module)
next_with_idle_timeoutbackstop that caps the freeze at ~295s instead of forever — but still ships no HTTP/2 keepalive ping, so this timeout fix remains valuable there too.Verification
cargo clippy --all-targets -D warningsclean,cargo fmtclean.watch_configunit tests (invariant: default < 240s;bounded()sets the timeout; env override clamped).watcher::Config::default()call sites across 11 reconcilers converted; 3 now-dead imports removed.Note
This is a controller fix — running clusters pick it up via the new
kars upgrade(orkubectl rollout restart). Does not affect first-time deploys.