Controller: InferencePolicy/ToolPolicy watch goes stale on AKS — CR edits don't reach the ConfigMap without a controller restart

[pallakatos]

Summary

On AKS, the controller's reconcilers intermittently stop reflecting CR spec edits into their compiled ConfigMaps. Editing an InferencePolicy (e.g. kubectl patch / kars policy) bumps metadata.generation, but the controller keeps recompiling the pre-edit object on its 15s requeue; the ConfigMap only updates after a controller restart (which forces a fresh List).

This is separate from the router hot-reload bug fixed in #450 / v0.1.14 (that one is the router side; this is the controller side).

Evidence (live, kars-aks, 2026-06-24)

• Patched palkarstest-inference requirePromptShields: true→false; kubectl get confirmed false in etcd; controller logged InferencePolicyCompiled every 15s with a frozen version_hash/compiled_digest and the ConfigMap stayed true (rv unchanged) until rollout restart, after which it compiled false.
• Reconcile logs showed generation: 3 (fresh metadata) yet a frozen compiled hash — i.e. reconciling fresh metadata against a stale reflector store spec.
• Leader election healthy (single leader; standby = 0 reconciles); no panics; no watch errors logged; lease renewing normally. The reconcile loop ran in bursts with multi-minute gaps.
• Propagation was sometimes minutes, sometimes never-without-restart — intermittent.

Likely cause

All controllers use watcher::Config::default() (kube-rs 3.1.0) with no watch timeout. On AKS, konnectivity/LB can silently drop an idle watch connection (no FIN/RST); kube-rs doesn't detect the dead stream and never re-lists, so the reflector store freezes while the 15s requeue keeps reconciling stale objects.

Suggested investigation / fix

• Add a bounded watch timeout (e.g. watcher::Config::default().timeout(~290)) and/or TCP keepalive / HTTP2 ping on the kube client so a silently-dropped watch is re-established (relist) within a bounded window.
• Consider a reconcile-activity watchdog that relists (or restarts) if no events arrive for N minutes.
• Repro with RUST_LOG=kube=debug on AKS to capture the watch lifecycle.

Impact

kars policy / InferencePolicy / ToolPolicy / EgressApproval live edits may not apply until a controller restart. Does not affect first-time deploys (the first reconcile compiles correctly from the initial List).

[denial123789]

This is a useful failure mode to make explicit for agent-runtime operators.

For ToolPolicy / InferencePolicy style control planes, I would treat "policy freshness" as part of the runtime safety contract, not just a controller implementation detail.

One validation angle that may help: alongside the watch timeout / relist fix, add a small conformance case that records:

• CR generation observed by the controller
• compiled policy digest / ConfigMap resourceVersion
• max propagation time from policy edit to effective runtime config
• router/runtime acknowledgement of the new digest

That would make stale reflector behavior visible as a bounded freshness SLO instead of only surfacing through a restart workaround.

[pallakatos]

Fixed in #456 (merged to main).

Root cause (refined): watcher::Config::default() isn't actually "no timeout" — kube-rs sends timeoutSeconds=290 by default. The problem is the value: Azure Standard LB's idle timeout is 240s, so a quiet watch is silently dropped (no FIN/RST) at 240s, before the server's 290s close. kube-rs 3.1.0 then polls the dead stream with a bare stream.next().await (no client-side idle deadline) and hangs forever → the reflector store freezes.

Fix: a shared watch_config::bounded() that sets the watch timeout to 200s (< the 240s LB idle; kube-rs rejects ≥295s), applied to every primary and secondary (.owns/.watches) watch across all 11 CRD reconcilers so it can't diverge again. The server now closes each watch at 200s — that traffic prevents the connection from ever sitting idle for 240s, and the watcher re-Lists within a bounded window. Overridable via KARS_WATCH_TIMEOUT_SECS (clamped 30..=600).

Researched alternatives, documented in the module:

• kube-rs 4.0 (PR #1945) adds a client-side next_with_idle_timeout backstop that caps the freeze at ~295s instead of forever — but still ships no HTTP/2 keepalive ping, so this timeout fix stays valuable there too.
• The client-go-parity gold standard is HTTP/2 keepalive ping on the client (~45s detection); it needs a custom hyper client stack + deps and is tracked as a defense-in-depth follow-up.

---

@denial123789 — good call on treating policy freshness as a bounded SLO rather than only an implementation detail; that's the right framing. Partial status on the conformance signals you listed: the controller already surfaces CR generation (observedGeneration) and the compiled digest (version_hash/compiled_digest), and #450 added router-side digest-propagation coverage. The genuinely new piece — a conformance case asserting a bounded max edit→effective propagation time + runtime ack — is a worthwhile follow-up and exactly the kind of test that would have caught this regression directly; I've noted it for the conformance suite.