feat(local): --nv-sentinel Tilt consumer for GPU health monitoring and fault remediation - #716
Draft
giuliocalzo wants to merge 35 commits into
Draft
feat(local): --nv-sentinel Tilt consumer for GPU health monitoring and fault remediation#716giuliocalzo wants to merge 35 commits into
giuliocalzo wants to merge 35 commits into
Conversation
giuliocalzo
marked this pull request as draft
August 24, 2026 10:22
NVSentinel's GPU Health Monitor polls nv-hostengine through the GPU Operator's standalone DCGM Service, which the shared operator values disable because nothing in the plain --gpu-operator scenario polls one. Layer it back on through the extra_values hook, and turn on the mock's dynamic metrics so a heated GPU reads differently from its idle siblings. The flag implies --gpu-operator and refuses --fgo and --compute-domain, which either remove the DCGM operand or bring their own cluster shape. Signed-off-by: Giulio Calzolari <gcalzolari@nvidia.com>
… true tilt up -- --nv-sentinel --fgo died with "--fgo is mutually exclusive with --gpu-operator", naming a flag the user never passed: --nv-sentinel implies --gpu-operator, so the pre-existing --fgo check fired first and the nv-sentinel-specific message below it was unreachable. Check the nv-sentinel pair before the --fgo pair. Two comments also asserted things the code around them contradicts. The DCGM overlay said the baseline disables the standalone DaemonSet because nothing polls one; the baseline in fact says the mock NVML cannot support DCGM at all, so say that and say why this overlay overrides it anyway. And disjoint values keys were offered as proof the nv-sentinel and observability overlays compose, which does not follow: dcgm.enabled=true reroutes dcgm-exporter onto the shared nvidia-dcgm:5555 hostengine, so record the coupling instead. The evaluated Tiltfile result is byte-identical either way; nothing deployed changes. Signed-off-by: Giulio Calzolari <gcalzolari@nvidia.com>
The ordering is load-bearing but was silent about it: --nv-sentinel implies --gpu-operator, so moving these checks below the --fgo pair silently restores the bug where the failure names a flag the user never passed. Signed-off-by: Giulio Calzolari <gcalzolari@nvidia.com>
NVSentinel needs change streams, so its datastore must be a replica set, and it talks TLS to it. The chart's bundled Bitnami MongoDB is amd64-only and cannot start on Apple Silicon, so run the official multi-arch image as an external datastore with a cert-manager-issued certificate, as the standalone demo does. The Namespace and the MONGODB_URI Secret move into the manifest: run.sh created both with imperative kubectl calls, which Tilt has no equivalent for. The URI carries no credentials. Signed-off-by: Giulio Calzolari <gcalzolari@nvidia.com>
Bound the mongodb-ext-rs-init Job with activeDeadlineSeconds: its two wait loops are unbounded, so a mongod that answers but never reaches PRIMARY hung the Job forever and `tilt ci` waited on it instead of failing. Split _CERT_MANAGER_NAMESPACE out of _CERT_MANAGER_RELEASE, order the cert-manager resource behind nvml-mock like the other consumers do (which retires an unused parameter), correct the crds.enabled comment, and drop install()'s unused return value along with the header sentence promising it. Also record why the MongoDB readiness probe stays a liveness check: asserting a writable PRIMARY there deadlocks, because Tilt withholds the Job that runs rs.initiate() until the pod it initialises is already Ready. Signed-off-by: Giulio Calzolari <gcalzolari@nvidia.com>
Ports the demo's values unchanged, comments included: they are the only record of why five non-default options are load-bearing here, from the thermal watch that only fires with store-only off to the circuit breaker that would otherwise halt recovery on a two-worker cluster. The release deliberately does not pass --wait. Its collection-setup Job is a post-install hook and the DB-consuming pods cannot go Ready before it runs, while Helm runs post-install hooks only after --wait is satisfied, so --wait deadlocks the release against its own hook. A separate nvsentinel-ready step waits for the Job and for the DCGM Service to have ready endpoints, then restarts the pods that raced them, instead of leaving them to time out an exponential backoff. Signed-off-by: Giulio Calzolari <gcalzolari@nvidia.com>
NVSentinel's GpuThermalMarginWatch arms only once it has the GPU's slowdown T.Limit offset (NVML field 194), which real hardware reports on Ada and later only and the mock gates on architecture the same way. On the default a100 profile the whole stack came up green with the watch inert, logging "missing slowdown TLIMIT threshold metadata" for every GPU and writing no thermal verdict at all — the silent no-op this consumer exists to rule out. So --nv-sentinel now defaults to h100, the profile the standalone demo pins, and rejects the two pre-Ada profiles instead of simulating a fault nothing can see. --multi-gpu-profile is rejected for the same reason: its fleet is exactly a100 plus t4. Signed-off-by: Giulio Calzolari <gcalzolari@nvidia.com>
Heats one mock GPU past its hardware slowdown limit and fails unless NVSentinel cordons the node and the GPU workload lands on the healthy worker, so the detect-and-remediate path is asserted rather than eyeballed. The temperature is derived from the profile the mock actually loaded instead of hardcoded. Slowdown thresholds range from 87C to 93C across profiles and shutdown sits only 3-5C above, where the mock clamps, so a fixed value would silently fail to trip anything on an l40s worker. The run also cannot pass on residue: it clears every override on every mock node and waits for the whole GPU fleet to be schedulable before injecting. Clearing only the target is not enough — the target follows the workload, so it alternates between runs, and the node the previous run left hot and cordoned is exactly the one this run needs as a drain destination. The workload drops the demo's nvml-mock-gpu nodeSelector. Requesting nvidia.com/gpu already keeps it on the workers, since the device plugin does not tolerate the control-plane taint. Signed-off-by: Giulio Calzolari <gcalzolari@nvidia.com>
The resource `tilt ci` blocks on could report success over a stack that detects nothing. Four reachable paths, all closed here: - The DCGM gate broke on any ready endpoint, so on a two-worker cluster it could restart both health monitors while one node's DCGM was still pulling, handing that monitor a dead hostengine it then reports Ready over. It now requires as many ready endpoints as the DCGM DaemonSet has scheduled pods, and a desired count of 0 no longer satisfies it. - An empty pod list read as "all Ready", which is also what a get landing in the window the pod delete opens returns. It now keeps polling. - The DCGM probe swallowed every error class through `2>/dev/null` and `|| true` on kubectl and jq alike, so an RBAC denial cost the full 20 minutes and then blamed the image pull. jq is gone (jsonpath instead, on a path that had no other jq dependency), kubectl's stderr is kept, and a run of consecutive API errors fails with the error text — while a NotFound, the one class waiting does fix, still waits. - Nothing asserted that GpuThermalMarginWatch actually armed. Every check above passed on the a100 profile while the watch was inert. The script now requires the field-153 watch line from every monitor pod and the absence of "missing slowdown TLIMIT threshold metadata" over a full fresh poll cycle, so green means armed rather than scheduled. Also fail immediately on a Failed hook Job instead of waiting out the timeout, and clamp the progress stride so POLL_INTERVAL_S above 60 prints progress instead of a division-by-zero error. Signed-off-by: Giulio Calzolari <gcalzolari@nvidia.com>
The guardrail rejected a100 and t4 by name, so any profile added later was allowed by default — and a new pre-Ada profile would have brought back the green-but-blind stack the guardrail exists to prevent. Invert it into an allow-list of the Ada-or-later profiles: an unknown or misspelled profile now fails with the same message instead of installing. Signed-off-by: Giulio Calzolari <gcalzolari@nvidia.com>
install() returned the release name and nothing consumed it. The values header claimed "five options here are non-default", which a reader can disprove in ten seconds — the file overrides many more keys than five; the five are the ones whose behaviour differs from the chart defaults. Signed-off-by: Giulio Calzolari <gcalzolari@nvidia.com>
The drain half of this scenario could pass without a drain. The workload's node was read before a pre-injection wait that may block 300s, and phase 2 only asked whether the workload was Running on some node other than the target — so a pod that was never on the target satisfied it on the first poll, with no eviction, and the script printed that the workload had moved. The target is now resolved after that wait, from the node the workload is actually on, and phase 2 requires the pod NAME to change as well: what is asserted is that the pod which was Running on the target when the fault went in is gone and its replacement runs elsewhere. Resolving the target after the wait rather than failing on a stale reading keeps an interrupted previous run recoverable, which is what the fleet-wide reset exists for; the premise is still asserted, and a workload that never becomes Running now fails before anything is injected instead of leaving a GPU pinned hot. The threshold reads were also unreachable diagnostics: `threshold()` is a pipeline under `pipefail`, so a missing key or a failed exec aborted the script on the assignment before the guard that would explain it, and `2>/dev/null` hid kubectl's own error. Both now reach the guard with the error text intact. Also: every post-injection failure names the GPU left pinned and the command that clears it, the informational `kubectl get nodes` can no longer fail the run after the assertions passed, `diagnose` reports the node each condition belongs to, and the exec form lives in one `mock_ctl_on` helper. Signed-off-by: Giulio Calzolari <gcalzolari@nvidia.com>
The scenarios hardcode `default` as the workload namespace, while the manifest left it implicit and so followed whatever namespace the applying context selects. Making it explicit keeps the assertions looking where the workload actually is. Signed-off-by: Giulio Calzolari <gcalzolari@nvidia.com>
An evicted pod keeps status.phase=Running until it is actually gone, so a drain still in flight offered this scenario a pod under deletion as its target. Its replacement is by then already Running on the other node, which would satisfy the phase 2 eviction check without this run having evicted anything. Excluding pods with a deletionTimestamp closes that. Signed-off-by: Giulio Calzolari <gcalzolari@nvidia.com>
Clears the temperature override on every cordoned GPU node and fails unless NVSentinel uncordons it, with no DCGM restart anywhere. That self-clearing is the reason this scenario is driven through the thermal margin rather than a latched XID or ECC fault, which would stay latched until nv-hostengine was bounced. Refuses to run when nothing is cordoned instead of asserting "the node is schedulable" against a node that was never quarantined, which would pass without exercising the recovery path at all. Signed-off-by: Giulio Calzolari <gcalzolari@nvidia.com>
local/README.md carries the flag and the two triggers; the consumer README carries the reference detail a reader needs once the stack is up: what lands in the cluster, what each file is for, the six NVSentinel options that make the loop work, and the behaviours that look like bugs and are not — chiefly the pods that crash-loop until the post-install hook creates the MongoDB collections, and why the release must not pass --wait. The demo index and the demo's own README now point at the Tilt flag, and it points back at them, so neither way of running this is a dead end. Signed-off-by: Giulio Calzolari <gcalzolari@nvidia.com>
The baseline GPU Operator values justified dcgm.enabled=false with "DCGM is not supported by the mock NVML (requires full driver stack)". The --nv-sentinel consumer disproved that: pointed at the mock driver root, nv-hostengine comes up and serves the nvidia-dcgm Service on :5555. The real reason it stays off by default is that nothing else polls a standalone hostengine — dcgm-exporter embeds its own — so say that instead. Comment only; the rendered values are unchanged. Signed-off-by: Giulio Calzolari <gcalzolari@nvidia.com>
recover-node's pin guard matched any temperature_gpu_c override at any value, so a node carrying only a cold pin from a by-hand session passed a guard meant to prove a hot one — and the uncordon it went on to assert was then somebody else's. Compare the pinned value against the profile's own slowdown_threshold_c instead, read through a second copy of quarantine-node's reader rather than a shared library, since scenario scripts here are self-contained. Proven on the live cluster: with a node genuinely quarantined and its pin replaced by a cold 40C one, the previous script exits 0 claiming the recovery, this one exits 1 in a second naming the 40C pin and the 87C threshold. Also: state the recovery time this path measured rather than Task 4's, hedge the fleet-check message for a node re-quarantined mid-poll, drop the unchecked "unmoved by the recovery" claim from a header that only reports placement, give both copies of cordoned_gpu_nodes the same separator, and say why recover-node shares the workload dependency. Signed-off-by: Giulio Calzolari <gcalzolari@nvidia.com>
…iceMonitor The overlay lowers the ServiceMonitor interval to 5s but left the chart's default scrapeTimeout of 10s, and prometheus-operator rejects any endpoint whose scrapeTimeout exceeds its interval. Measured under `tilt ci -- --nv-sentinel --observability`: the ServiceMonitor is created, every Tilt resource goes green, and the operator logs `skipping object ... scrapeTimeout "10s" greater than scrapeInterval "5s"` with an InvalidConfiguration event — so Prometheus had no dcgm-exporter target at all and every GPU panel was empty. With scrapeTimeout at 5s both exporter endpoints scrape healthy and DCGM_FI_DEV_GPU_TEMP reports all 16 mock GPUs. Signed-off-by: Giulio Calzolari <gcalzolari@nvidia.com>
…armed A pod keeps status.phase=Running while it terminates, so the pod restart earlier in the script leaves doomed pods matching the field selector for a few seconds. One was selected, went away during the 20s log observation, and failed an otherwise healthy bring-up on `pods "gpu-health-monitor-dcgm-4.x-85b7v" not found` rather than on anything about the watch. Excluding them makes the DaemonSet's replacement the pod that is checked, and the desired-count guard already turns "not Running yet" into a wait. Signed-off-by: Giulio Calzolari <gcalzolari@nvidia.com>
Every command the README prints was run at this HEAD. Four claims did not survive that: - detection is 10-21s, not ~15s, and the eviction is in the same second or the next 5s poll, not always the same second; - `tilt trigger` needs a live `tilt up` session, and on the `tilt ci` path it fails on a refused connection to localhost:10350; - re-running the rs-init Job is not the whole MongoDB recovery: the HealthEventsDatabase collections go with the emptyDir, and until the chart's setup hook re-runs, fault-quarantine and node-drainer crash-loop on `no collection with name HealthEvents`; - adding --observability to a stack whose nvidia-dcgm is already serving restarts no dcgm-exporter pod, so the documented crash-loop belongs to the cold DCGM pull rather than to the flag pair. Adds the drain/monitoring interaction that composed run exposed — node-drainer's userNamespaces "*" evicts Prometheus if it sits on the quarantined worker, and its storage is not persistent — plus the composed bring-up cost and recover-node's narrowed pin guard. Signed-off-by: Giulio Calzolari <gcalzolari@nvidia.com>
This reverts commit 355ec5d. Signed-off-by: Giulio Calzolari <gcalzolari@nvidia.com>
…ter ServiceMonitor" This reverts commit 10abc97. Signed-off-by: Giulio Calzolari <gcalzolari@nvidia.com>
nvsentinel-ready asserted the thermal watch was armed by the ABSENCE of the per-GPU "missing slowdown TLIMIT threshold metadata" message. A monitor that enumerated zero GPUs produces that same absence: its per-GPU evaluation loop iterates an empty list, so nothing is evaluated and nothing is logged, while the armed line is still printed and the DCGM endpoint gate is still satisfied by a healthy hostengine with no devices behind it. Every check passed over a fleet monitored by nothing. Pair the absence with a positive count read from the monitor's own "dcgm gpu_id are [...]" init line, cross-checked against the GPUs the node advertises. Also make the file executable, like every other script under local/. Signed-off-by: Giulio Calzolari <gcalzolari@nvidia.com>
mkdocs builds only docs/, so a relative link out to local/ resolves against nothing and `mkdocs build --strict` fails the docs job. Point at the file on GitHub, matching how the demo page already links local/observability and local/README. Signed-off-by: Giulio Calzolari <gcalzolari@nvidia.com>
--topograph implies --compute-domain, whose 4-worker cluster NVSentinel is untested on, so the pair would otherwise land on the --compute-domain guardrail and name a flag the user never passed. Signed-off-by: Giulio Calzolari <gcalzolari@nvidia.com>
Ada-and-later profiles are the ones that report the slowdown T.Limit thermal margin the thermal-margin consumers key on, so making gb300 the default stops the out-of-the-box shape from excluding them: --nv-sentinel no longer has to override the default, and a plain `tilt up` can drive the same watch. Pass --gpu-profile a100 for the previous shape; the --multi-gpu-profile fleet is untouched and stays a100 + t4. Documented thresholds follow the new default: gb300 slows down at 90C and shuts down at 95C, so quarantine-node injects 93C rather than h100's 90C. Signed-off-by: Giulio Calzolari <gcalzolari@nvidia.com>
quarantine-node took the first of however many pods matched the workload selector, and phase 2 then passes as soon as a pod with a different name is Running elsewhere. A second replica satisfies that before node-drainer evicts anything, so the drain assertion silently weakens to "the workload is somewhere else" — green without a drain. The workload ships replicas: 1, so this asserts an invariant the selection already depended on; it is polled rather than failed outright because a rollout can briefly surge to two Running pods. Signed-off-by: Giulio Calzolari <gcalzolari@nvidia.com>
The control-plane section gave a false reason for a true conclusion: the node DOES carry nvidia.com/gpu.present=true and driver.installed=true, and NVSentinel's metadata-collector schedules there on exactly those labels. What keeps nvidia.com/gpu off it is narrower — the device plugin tolerates only nvidia.com/gpu:NoSchedule, not the control-plane taint. Neither README covered what happens when --observability's inject-thermal runs under --nv-sentinel. Its fixed 90C pin is chosen against each profile's shutdown threshold, but GpuThermalMarginWatch keys on the lower slowdown threshold and fails once the pin is strictly above it: on h100 (87C) the observability trigger therefore cordons and drains the worker and leaves it that way, since it keeps the pin deliberately. Documented on both sides, per profile, with the recovery. Signed-off-by: Giulio Calzolari <gcalzolari@nvidia.com>
Applying a Certificate only asks cert-manager to issue one, so shipping the Certificates in the same Tilt resource as the StatefulSet that mounts the resulting Secret starts the pod against a Secret that is not there yet. A live bring-up shows the cost: `FailedMount ... secret "mongodb-server-tls" not found`, after which kubelet retries on a backoff that outlasts the issuance — recovered only because a 2m33s image pull happened to cover the gap. The Certificates move into their own mongodb-certs resource, a mongodb-certs-ready wait blocks on both of them reaching Ready, and mongodb-ext orders behind that. Both Certificates, not just the server one, because mongodb-ca-issuer signs with the CA Secret and NVSentinel mounts it as its trust root. Note for anyone with a running session: this moves objects between Tilt resources, including the Namespace, so the first reload tears the nvsentinel namespace down and rebuilds it. One time only, and Tilt does not cascade the rebuild on its own — trigger mongodb-certs, mongodb-ext, mongodb-ext-rs-init and nvsentinel in order, or restart the session. Signed-off-by: Giulio Calzolari <gcalzolari@nvidia.com>
The comment credited fault-quarantine and "the analyzer", but this deployment sets healthEventsAnalyzer.enabled=false. Measured against the running stack, the two open change streams on HealthEventsDatabase.HealthEvents belong to fault-quarantine and node-drainer. Also states why --replSet cannot simply be dropped, which the comment left as an assertion: change streams are derived from the oplog, so a standalone mongod answers a watch with Location40573 and the pipeline would store health events while reacting to none of them. Signed-off-by: Giulio Calzolari <gcalzolari@nvidia.com>
The comment claimed tls.enabled had to be true because the chart renders it true regardless via a `| default true`. That default exists, but only on the tls block inside the rendered datastore.yaml, and no pod mounts that file. The components read the setting from the nvsentinel-datastore-config ConfigMap through envFrom, where tls.enabled: false does render MONGODB_TLS_ENABLED="false" — the chart documents exactly that for external datastores without TLS. So TLS here is a deliberate choice to match a real deployment, not a constraint. Saying otherwise would send the next reader looking for a chart bug to work around instead of a decision to weigh. Signed-off-by: Giulio Calzolari <gcalzolari@nvidia.com>
The one-shot rs-init Job could not be combined with a readiness probe that asserts a writable primary: the Job depended on mongodb-ext being Ready, so a probe waiting for rs.initiate() withheld the Job that performs it. That left the pod Ready on a ping alone, and a recreated pod — emptyDir, so no replica set config survives — came up uninitialised behind an all-green UI. Running rs.initiate() in a postStart hook inverts the dependency: the hook runs inside the container that needs initiating, so the probe is free to require db.hello().isWritablePrimary, and the replica set now re-initiates with the pod instead of needing a Job re-triggered by hand. The hook must resolve the pod's own service DNS name, which is not published while postStart blocks the container from reaching Running, so rs.initiate() failed NodeNotFound and cost a restart plus ~3.5 minutes. A hostAliases entry mapping mongodb-ext.nvsentinel.svc.cluster.local to 127.0.0.1 lets mongod recognise itself immediately: measured Ready in 8 seconds with zero restarts. TLS stays on, since it mirrors the standalone demo and a real deployment. Plaintext is possible but takes two settings, not one: tls.enabled: false only renders MONGODB_TLS_ENABLED, while each component separately defaults clientCertMountPath to a real path and its binary reads ca.crt from there before connecting — with no CA mounted it logs "Failed to read CA certificate" for 360s and exits 1 while reporting Ready throughout. Both files record that trap. Signed-off-by: Giulio Calzolari <gcalzolari@nvidia.com>
Main now keeps nvml-mock on labelled workers only (NVIDIA#724), so three comments here justified their design with a control-plane mock pod that no longer exists. The designs are unchanged and still correct — the scenarios and the workload key on the advertised nvidia.com/gpu resource, which the device plugin never publishes on a control-plane it cannot tolerate — only the reasoning needed correcting. The DCGM enumeration example also implied eight GPUs, where the default gb300 is now a 4-GPU NVL72 compute tray (NVIDIA#727) and enumerates [0 1 2 3]. Signed-off-by: Giulio Calzolari <gcalzolari@nvidia.com>
giuliocalzo
force-pushed
the
feat/nv-sentinel-tilt-consumer
branch
from
August 27, 2026 08:24
405636b to
6e0f871
Compare
The mock pod gained a node-agent sidecar, so every `kubectl exec` here printed `Defaulted container "nvml-mock" out of: nvml-mock, node-agent` and depended on container ordering to reach nvml-mock-ctl and the rendered profile, neither of which exists in the sidecar. Naming the container makes the dependency explicit and drops the warning from otherwise clean scenario output. The two hints that tell a reader how to clear a pin by hand name it too, since a copy-pasted command should not need the same reasoning. Also corrects the DCGM enumeration example to the format the monitor actually logs, `[0, 1, 2, 3]`, verified against a live gb300 run. Signed-off-by: Giulio Calzolari <gcalzolari@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Ports the standalone
docs/demo/nv-sentinel/demo into the Tilt local-dev environment as a--nv-sentinelconsumer, so the whole NVSentinel remediation loop runs on a Kind cluster with no GPUs:NVSentinel runs unmodified against the mock GPUs through real NVIDIA DCGM. Two manual triggers assert the loop end to end rather than leaving it to be eyeballed:
quarantine-nodeheats a GPU past its hardware slowdown limit and asserts the node is cordoned and the GPU workload actually evicted and replaced elsewhere (~10-21s);recover-nodecools it and asserts the node comes back withGpuThermalMarginWatchhealthy and no DCGM restart (~10-16s).The standalone demo is kept as-is for use outside Tilt — this consumer skips its dedicated cluster and per-worker
nvidia-container-toolkitinstall, since the shared Kind node image bakes both in.The diff is confined to
local/nv-sentinel/plus the rootTiltfilewiring and docs; no existing consumer's files are touched.One repo-wide change, called out for review
The default
--gpu-profilemoves froma100togb300(Tiltfile, one line, plus the docs that quoted the old default). Ada-and-later profiles are the ones that report the slowdown T.Limit thermal margin the thermal-margin consumers key on, so the default no longer excludes them and a plaintilt upcan drive the same watch.--gpu-profile a100restores the previous shape, and the heterogeneous--multi-gpu-profilefleet is untouched (still a100 + t4). Flagging it because it is the only change here that affects sessions that never pass--nv-sentinel; happy to drop it and keep the default local to the flag if you would rather.Notable decisions
--gpu-profiledefaults togb300and pre-Ada profiles are rejected.GpuThermalMarginWatcharms only from the GPU's slowdown T.Limit offset, which real hardware reports on Ada and later only — and the mock gates it identically. Ona100/t4every Tilt resource goes green while the watch stays inert, so a heated GPU is detected by nothing on a stack that looks perfectly healthy. The allow-list (h100,l40s,b200,gb200,gb300) fails closed.--wait, which would deadlock: Helm runs post-install hooks only after--waitis satisfied, and the pods--waitis waiting on cannot become Ready until the hook creates their MongoDB collections.nvsentinel-readyis the wait instead, and it asserts the thermal watch is armed rather than merely scheduled — a greengpu-operatorresource does not mean DCGM is reachable.mongodb-storesubchart uses Bitnami images, which are amd64-only and run Bitnami-specific startup scripts — unusable on Apple Silicon.nvsentinel.values.yamloverrides carries its reason inline, including two that must stay at their defaults on real clusters (circuitBreaker,node-drainermode).A pre-existing
--observabilitybug found along the way, deliberately not fixed here--nv-sentinelcomposes with--observability, and verifying that combination surfaced a bug that predates this branch and affects plain--observabilitytoo: the overlay lowers the dcgm-exporter ServiceMonitor'sintervalto 5s but leaves the chart's defaultscrapeTimeoutat 10s, and prometheus-operator rejects any endpoint whose timeout exceeds its interval. It logsskipping object ... scrapeTimeout "10s" greater than scrapeInterval "5s"with anInvalidConfigurationevent, but the ServiceMonitor still exists and every Tilt resource still goes green — so the only symptom is empty GPU panels. SettingscrapeTimeout: 5sfixes it: both exporter endpoints then scrape healthy andDCGM_FI_DEV_GPU_TEMPreports all 16 mock GPUs.I had that fix in this branch and have taken it out, to keep this PR scoped to
local/nv-sentinel/. It belongs in its own PR where the observability owner will see it. Happy to open that, or leave it to whoever owns that consumer — say which you'd prefer. Until it lands,--nv-sentinel --observabilitybrings the stack up correctly but the Grafana GPU panels stay empty.Test plan
Verified live on a 3-node Kind cluster (1 control-plane, 2 workers):
tilt ci -- --nv-sentinelconverges; cold ~8-15 min (dominated by a ~14 mingpu-health-monitorimage pull), warm under a minutetilt ci -- --nv-sentinel --observabilityconverges (Prometheus and Grafana healthy; GPU panels empty pending the ServiceMonitor fix above)GpuThermalMarginWatchverdictsquarantine-nodethenrecover-nodepass, repeatedly and back-to-backrecover-noderefuses a healthy cluster, a hand-cordoned node, and a quarantined node with no hot pin;quarantine-nodefails rather than reporting a reschedule that did not happen, and refuses to inject unless exactly one workload pod is Running (a second replica would satisfy the eviction assertion on its own)--nv-sentinelimplies--gpu-operator; rejects--fgo,--compute-domain,--topograph,--multi-gpu-profileand pre-Ada profiles, each with a message naming the reasonmake testandmake helm-testspass;golangci-lintclean (make lintfails only on pre-existing govulncheck CVEs in vendored deps, unrelated to this change)make docs(mkdocs--strict) passes with the branch's docs applied on top of currentmainKnown trade-off, documented in the README: deleting the
mongodb-extpod leaves an uninitialised replica set behind a green Tilt UI (emptyDirdata + one-shot init Job). Recovery istilt ci -- --nv-sentinel. The obvious fix — a readiness probe asserting PRIMARY — was tried and reverted because it deadlocks against the Job that performs the initialisation.