fix(power): DCGM CPU power is the CPU rail; publish rail + SysIO breakdown, prefer ACPI in auto

docs/cpu-power-telemetry.md

@@ -104,10 +104,66 @@ The exporter binary itself decides ACPI vs. DCGM per its own `--source` flag:
   it) a `total`-kind rail per socket. Domain names vary by platform (e.g.
   "Grace Power Socket 0" vs. a generic "Total Power socket 0", some suffixed
   with "in uW"); the exporter classifies all known variants into these kinds.
-- **`dcgm`** — reads DCGM CPU entity power directly, one already-aggregated
-  value per socket.
-- **`auto`** (default) — tries DCGM first, falls back to ACPI when DCGM is
-  unavailable or reports no CPU entities.
+- **`dcgm`** — reads the DCGM CPU-entity power fields. DCGM has no CPU power
+  backend of its own: its sysmon module reads the same ACPI hwmon channels by
+  `power1_oem_info` label, so each field *is* one ACPI rail:
+
+  | DCGM field | hwmon label it reads | srtctl rail | CSV column |
+  | --- | --- | --- | --- |
+  | 1130 `DCGM_FI_DEV_CPU_POWER_WATTS` | `CPU Power Socket N` | `cpu_rail` | `power_w` **and** `cpu_rail_w` |
+  | 1132 `DCGM_FI_DEV_SYSIO_POWER_UTIL_CURRENT` | `SysIO Power Socket N` | `soc` | `soc_w` |
+  | 1131 `DCGM_FI_DEV_CPU_POWER_LIMIT_WATTS` | `Grace Power Socket N` (**cap** only) | — | — |
+  | 1133 `DCGM_FI_DEV_MODULE_POWER_UTIL_CURRENT` | `Module Power Socket N` | — (unverified; excluded) | — |
+
+  No DCGM field reports the `Grace Power Socket N` envelope's draw. Field
+  1133 does read a usage file, but what "Module" spans on GB200/GB300
+  (Grace alone, or the Grace+Blackwell superchip) has not been measured on a
+  live node; it is excluded until `dcgmi dmon -e 1130,1131,1132,1133 -i
+  cpu:0` alongside `cat /sys/class/hwmon/*/device/power1_oem_info` settles
+  it. So
+  **DCGM-mode `power_w` is the CPU rail, roughly half of ACPI-mode `power_w`**
+  (about 53 W vs 100 W per socket on GB200 reference runs). The energy report
+  attaches a "CPU rail only" warning to DCGM-sourced runs; do not compare their
+  CPU figures with ACPI-mode runs as like-for-like. The exporter publishes one
+  `cpu_power_dcgm_watts{socket,field_id}` sample per field; the collector files
+  1130 as `power_w` (as it always has) and also as `cpu_rail_w`, and 1132 as
+  `soc_w`. Older exporter builds publish 1130 alone without a `field_id`
+  label; the collector treats those samples as 1130.
+  Source: NVIDIA/DCGM `modules/sysmon/DcgmSystemMonitor.cpp` (label → file
+  map) and `modules/sysmon/DcgmModuleSysmon.cpp` (field id → getter).
+- **`auto`** (default) — walks a fixed ladder at startup, most informative
+  source first, and each source must prove itself before it is used. Every
+  step down is logged at WARN in the exporter's `.out` file with the reason:
+
+  1. **ACPI, live** — `power_meter` hwmon sensors were discovered *and* at
+     least one **socket-total** channel (`Grace Power Socket N` / `Total
+     Power socket N`) reads a positive value. Discovery alone is not enough:
+     a node can expose channels that never report, and publishing them would
+     integrate to 0 J and pass for a measurement. Only the total counts
+     because it is what becomes `power_w`; a live component rail next to a
+     dead envelope would otherwise vouch for a socket whose power can never
+     be published. If no total reads positive, the probe is repeated once
+     after 1 s (hwmon averages can read 0 on the first poll after boot)
+     before ACPI is declared dead.
+  2. **DCGM, fields 1130 + 1132** — CPU rail and SysIO.
+  3. **DCGM, field 1130 alone** — when this libdcgm refuses 1132 for CPU
+     entities (older release, or a sysmon without SysIO). 1130 alone is the
+     floor every earlier exporter had; in `auto` DCGM is the last resort, so
+     failing here would mean no CPU power at all.
+
+     Either DCGM rung is accepted only if at least one CPU entity then
+     returns a positive field 1130 on a first read — that is the value filed
+     as `power_w`. SysIO alone does not make DCGM usable: a node where only
+     1132 reads would publish `soc_w` for the whole run with no `power_w`
+     and no socket rows. An all-`None` read usually means the embedded
+     daemon lacks hardware access (running without root while a system
+     dcgm-exporter holds the session).
+  4. **Exit non-zero** — no source could be established.
+
+  `--source acpi` and `--source dcgm` run only their own rungs (ACPI still
+  requires the liveness probe to pass; DCGM still tries 1130+1132 then 1130)
+  and exit non-zero instead of stepping down. The Python host collector
+  (`srtctl.core.cpu_power`) implements the same ladder.
 
 The exporter resolves this once at process startup and serves only one metric
 family (`cpu_power_dcgm_watts` or `cpu_power_acpi_watts`) for its lifetime.
@@ -122,12 +178,32 @@ readings if a scrape body ever contained both, since ACPI carries more detail.
 schema_version, timestamp_unix, hostname, source, sensor, socket_id, power_w, total_power_w
 ```
 
-- **`power_w`** — one sensor's power reading for that scrape. `sensor` names
-  look like `CPU0:cpuPowerUsageW` (ACPI) or a DCGM field label; granularity is
-  per-socket.
+- **`power_w`** — the socket's power for that scrape: the ACPI `total`
+  envelope (`sensor` = `CPU<n>:cpuSidePowerUsageW`) or DCGM field 1130
+  (`sensor` = `CPU<n>:cpuPowerUsageW`, which is the CPU rail — see above);
+  granularity is per-socket.
+- **`cpu_rail_w`, `soc_w`, `dram_w`** — component rails for the socket, blank
+  when the source has no such reading. ACPI fills whichever rails the firmware
+  exposes; DCGM fills `cpu_rail_w` (= `power_w`, field 1130) and `soc_w`
+  (field 1132) and leaves `dram_w` blank.
+
+  A rail that could not be read is **blank, never `0`**, whatever the cause:
+  the exporter fell back to watching 1130 alone (blank on every row, one WARN
+  at startup); DCGM returned a non-OK status for that (socket, field) on that
+  scrape; or the value was zero or non-finite (every reader -- the exporter,
+  the scrape parser and the host collector -- drops it, since `0` from these
+  files means "not measured", not "idle": a live Grace socket draws tens of
+  watts). The primary is
+  different: a socket whose `power_w` source (ACPI `total`, or field 1130) is
+  missing on a scrape produces **no row** for that socket rather than a row
+  with blank `power_w`. Component rails never substitute for it and are
+  never summed into `total_power_w`, so a missing `soc_w` changes nothing
+  else on the row. Consumers should treat a blank rail as "unknown", not as
+  a zero-watt rail.
 - **`total_power_w`** — the node-level total for that scrape, duplicated on
   every sensor row at the same `(hostname, timestamp_unix)`. In DCGM mode this
-  is the sum of the per-socket DCGM values. In ACPI mode it is **not** a sum of
+  is the sum of the per-socket field-1130 values (CPU rails; SysIO is never
+  added). In ACPI mode it is **not** a sum of
   the `cpu_rail`-, `soc`-, and `dram`-kind rails: whenever a `total`-kind
   channel exists for a socket, that channel alone is the total. Real hardware
   traces show the `total` rail at roughly 93-104W against `cpu_rail`+`soc`
@@ -200,8 +276,12 @@ are a separate, unrelated top-level config (`gpu_power_limits`) — not part of
 scraper design. Instead of an exporter plus a head-node poller, srtctl launches
 `python3 -m srtctl.core.cpu_power` directly on the bare host of every worker
 node. Each collector reads Linux ACPI `power_meter` hwmon channels (or DCGM CPU
-entity field 1130) itself, writes its own per-node CSV under
-`<storage_subdir>/nodes/`, and drops a ready marker. At teardown the head node
+entity fields 1130 and 1132) itself, writes its own per-node CSV under
+`<storage_subdir>/nodes/`, and drops a ready marker. Its `--source auto`
+walks the same ladder as the exporter (ACPI only if some sensor reads
+positive, then DCGM 1130+1132, then 1130 alone), logging each step down to
+the collector's `.out` file; the per-node `*.metadata.json` records which
+DCGM power fields were actually watched (`power_fields[].watched`). At teardown the head node
 (`CpuPowerTelemetrySession`, `src/srtctl/core/cpu_power_session.py`) merges the
 node CSVs into `<storage_subdir>/samples.csv` and writes `manifest.json`.
 
@@ -210,7 +290,7 @@ telemetry:
   enabled: true
   cpu_power:
     enabled: true              # presence alone is not enough; this flag turns the leg on
-    source: auto               # "auto" (ACPI then DCGM, best-effort) | "acpi" | "dcgm" (mandatory)
+    source: auto               # "auto" (live ACPI, then DCGM 1130+1132, then 1130) | "acpi" | "dcgm" (mandatory)
     sample_interval_seconds: 0.1
     startup_timeout_seconds: 30.0
     required: false            # true fails the job if the leg is not ready or not publishable
@@ -229,7 +309,7 @@ Differences from `cpu_power_exporter`:
   extra `timestamp_local` column, not in `power/cpu/`. The energy report
   (`python -m srtctl.analysis.power_energy_report <log_dir>`) discovers either
   location; when a run has both, pass `--cpu-samples <path>` to pick one.
-- **Per-socket utilization (DCGM source only).** Alongside power field 1130
+- **Per-socket utilization (DCGM source only).** Alongside the power fields
   the DCGM reader watches CPU entity fields 1100-1104 and appends five
   columns to every sample row: `cpu_util_total`, `cpu_util_user`,
   `cpu_util_nice`, `cpu_util_sys`, `cpu_util_irq`, reported by DCGM as a

docs/dcgm-4.7-runtime-support.md

@@ -66,9 +66,14 @@ or generated Slurm script.
    update, and read with `dcgmEntitiesGetLatestValues(..., flags=0)`. A live
    query returned DCGM status `-32` in the validated environment.
 8. **Keep power semantics explicit.** DCGM field 1130 is CPU-rail power, not the
-   full CPU-side socket total exposed by hwmon. Mark the manifest aggregate
-   scope as `cpu_rail_only`; do not silently compare or merge it with hwmon
-   `Total Power`.
+   full CPU-side socket total exposed by hwmon: DCGM's sysmon reads
+   `power1_average` of the hwmon channel labelled `CPU Power Socket N` for
+   1130 and `SysIO Power Socket N` for 1132, and uses `Grace Power Socket N`
+   only for the cap (1131) — NVIDIA/DCGM `modules/sysmon/DcgmSystemMonitor.cpp`,
+   `modules/sysmon/DcgmModuleSysmon.cpp`. Watch 1130 and 1132 together, file
+   1130 as `power_w` and `cpu_rail_w` and 1132 as `soc_w`, mark the manifest
+   aggregate scope as `cpu_rail_only`, and do not silently compare or merge it
+   with hwmon `Total Power`.
 9. **Clean up deterministically.** Unwatch fields, delete field/entity groups,
    shut down the embedded engine, and remove only the job-scoped extraction
    directory. Preserve the downloaded package identity and runtime provenance

docs/observability.md

@@ -136,8 +136,8 @@ Fields: [TelemetryConfig](schema-reference.md#telemetryconfig). `collect_interva
 
 `telemetry.cpu_power_exporter` is an independent, best-effort leg: its presence (not a separate `enabled` flag) turns CPU power collection on, and it can run with or without `dcgm_exporter` alongside it. On each worker node, srtctl launches a `cpu-power-exporter` process directly on the bare host (outside the model container, so it can read host power interfaces) and exposes it on `cpu_power_exporter.port`. It resolves the bundled Rust binary installed by `make setup` first, falling back to the ACPI-only Python stdlib exporter (`srtctl.core.cpu_power_exporter`) when that binary is absent. A head-node collector scrapes every worker's exporter on the shared `collect_interval_ms`/`request_timeout_seconds` cadence and writes per-sample rows plus a manifest under `<log_dir>/<storage_subdir>/cpu/` (`samples.csv`, `cpu_manifest.json`).
 
-Fields: [CpuPowerExporterConfig](schema-reference.md#cpupowerexporterconfig). `source: auto` tries DCGM first and falls back to ACPI; it has no effect on the Python fallback exporter, which is ACPI-only.
+Fields: [CpuPowerExporterConfig](schema-reference.md#cpupowerexporterconfig). `source: auto` uses ACPI when at least one discovered socket-total sensor reads a positive value (the envelope that becomes `power_w`; a 0 W reading from any channel is filed as missing, never as a sample), else DCGM fields 1130+1132 (CPU rail + SysIO), else 1130 alone if this libdcgm refuses 1132 — each step down logged at WARN. DCGM has no envelope field, so its `power_w` is ~half of ACPI's. The setting has no effect on the Python fallback exporter, which is ACPI-only. See [CPU power telemetry](cpu-power-telemetry.md) for the full ladder.
 
-`samples.csv` carries one row per sensor reading, with columns `schema_version, timestamp_unix, hostname, source, sensor, socket_id, power_w, total_power_w`. In ACPI mode, `total_power_w` is **not** a sum of the `cpu`- and `sysio`-kind rails; whenever a `grace`-kind channel exists for a socket, that channel alone is the node-level total (real hardware traces show `grace` at roughly 93-104W against `cpu`+`sysio` combined at roughly 53-58W for the same socket, i.e. `grace` measures the whole Grace SoC power boundary, not literally `cpu + sysio`). When no `grace` channel is present for a scrape, `total_power_w` is left blank for that row rather than guessed from the component rails; per-socket `power_w` values are always populated regardless. In DCGM mode, `total_power_w` is the single already-aggregate value DCGM reports per socket.
+`samples.csv` carries one row per sensor reading, with columns `schema_version, timestamp_unix, hostname, source, sensor, socket_id, power_w, total_power_w`. In ACPI mode, `total_power_w` is **not** a sum of the `cpu`- and `sysio`-kind rails; whenever a `grace`-kind channel exists for a socket, that channel alone is the node-level total (real hardware traces show `grace` at roughly 93-104W against `cpu`+`sysio` combined at roughly 53-58W for the same socket, i.e. `grace` measures the whole Grace SoC power boundary, not literally `cpu + sysio`). When no `grace` channel is present for a scrape, `total_power_w` is left blank for that row rather than guessed from the component rails; per-socket `power_w` values are always populated regardless. In DCGM mode, `power_w` is DCGM field 1130, which DCGM's sysmon reads from the `CPU Power Socket N` hwmon channel — the CPU rail, **not** the `grace` envelope (DCGM has no field for it); `total_power_w` is the sum of those rails, so DCGM-mode CPU power is roughly half of ACPI-mode CPU power and the energy report flags it as "CPU rail only". See [CPU power telemetry](cpu-power-telemetry.md) for the field→rail table.
 
 Because collection is always best-effort, there is no `required` knob for the CPU leg: a node that fails to expose its exporter (or fails to publish readings) simply produces gaps in `samples.csv`, and the run continues.

docs/schema-reference.md

@@ -482,8 +482,8 @@ Best-effort CPU power collection via the cpu-power-exporter binary.
 
 | Key | Type | Default | Description |
 |---|---|---|---|
-| `port` | int | `9405` | Port the exporter listens on and the head-node collector scrapes. |
-| `source` | one of `'auto'`, `'acpi'`, `'dcgm'` | `'auto'` | Power reading back-end passed through to the bundled Rust binary's own ``--source`` flag (``auto`` \| ``acpi`` \| ``dcgm``). ``auto`` tries DCGM first and falls back to ACPI when libdcgm.so is absent or reports no CPU entities. Has no effect when the Python stdlib fallback exporter is used instead of the binary -- that fallback is ACPI-only. |
+| `port` | int | `9405` | TCP port the per-node exporter listens on for ``/metrics``. |
+| `source` | one of `'auto'`, `'acpi'`, `'dcgm'` | `'auto'` | Power reading back-end passed through to the bundled Rust binary's own ``--source`` flag (``auto`` \| ``acpi`` \| ``dcgm``). ``auto`` tries ACPI first (the only source with the socket envelope) and falls back to DCGM when no ACPI power_meter hwmon sensors exist or no socket-total sensor reads positive on two probes; DCGM watches fields 1130+1132 and falls back to 1130 alone if this libdcgm refuses the pair, and is accepted only if some socket reads a positive 1130. DCGM mode reports field 1130 = the CPU rail (plus 1132 = SysIO), about half the ACPI envelope, and the energy report flags such runs as "CPU rail only". Has no effect when the Python stdlib fallback exporter is used instead of the binary -- that fallback is ACPI-only. |
 
 ### CpuPowerConfig
 
@@ -492,7 +492,7 @@ Host-side CPU power collection on every worker node.
 | Key | Type | Default | Description |
 |---|---|---|---|
 | `enabled` | bool | `False` | Master switch for this leg. Default: False. |
-| `source` | one of `'auto'`, `'acpi'`, `'dcgm'` | `'auto'` | ``auto`` tries ACPI then DCGM and is best-effort; naming ``acpi`` or ``dcgm`` explicitly makes that provider mandatory. |
+| `source` | one of `'auto'`, `'acpi'`, `'dcgm'` | `'auto'` | ``auto`` uses ACPI when some socket-total sensor reads a positive value, else DCGM fields 1130+1132, else 1130 alone (either DCGM rung accepted only if some socket reads a positive 1130; SysIO alone cannot vouch), logging each step down; naming ``acpi`` or ``dcgm`` explicitly makes that provider mandatory (it still proves itself, but exits instead of stepping). |
 | `sample_interval_seconds` | float | `0.1` | Read period on each node, in seconds. |
 | `startup_timeout_seconds` | float | `30.0` | How long to wait for every node's collector to publish its ready marker before giving up on readiness. |
 | `required` | bool | `False` | Fail the job when the leg does not become ready or does not produce a valid publication. |

docs/schema/recipe.schema.json

@@ -633,7 +633,7 @@
             "acpi",
             "dcgm"
           ],
-          "description": "``auto`` tries ACPI then DCGM and is best-effort; naming ``acpi`` or ``dcgm`` explicitly makes that provider mandatory.",
+          "description": "``auto`` uses ACPI when some socket-total sensor reads a positive value, else DCGM fields 1130+1132, else 1130 alone (either DCGM rung accepted only if some socket reads a positive 1130; SysIO alone cannot vouch), logging each step down; naming ``acpi`` or ``dcgm`` explicitly makes that provider mandatory (it still proves itself, but exits instead of stepping).",
           "default": "auto"
         },
         "sample_interval_seconds": {
@@ -665,7 +665,7 @@
       "properties": {
         "port": {
           "type": "integer",
-          "description": "Port the exporter listens on and the head-node collector scrapes.",
+          "description": "TCP port the per-node exporter listens on for ``/metrics``.",
           "default": 9405
         },
         "source": {
@@ -674,7 +674,7 @@
             "acpi",
             "dcgm"
           ],
-          "description": "Power reading back-end passed through to the bundled Rust binary's own ``--source`` flag (``auto`` | ``acpi`` | ``dcgm``). ``auto`` tries DCGM first and falls back to ACPI when libdcgm.so is absent or reports no CPU entities. Has no effect when the Python stdlib fallback exporter is used instead of the binary -- that fallback is ACPI-only.",
+          "description": "Power reading back-end passed through to the bundled Rust binary's own ``--source`` flag (``auto`` | ``acpi`` | ``dcgm``). ``auto`` tries ACPI first (the only source with the socket envelope) and falls back to DCGM when no ACPI power_meter hwmon sensors exist or no socket-total sensor reads positive on two probes; DCGM watches fields 1130+1132 and falls back to 1130 alone if this libdcgm refuses the pair, and is accepted only if some socket reads a positive 1130. DCGM mode reports field 1130 = the CPU rail (plus 1132 = SysIO), about half the ACPI envelope, and the energy report flags such runs as \"CPU rail only\". Has no effect when the Python stdlib fallback exporter is used instead of the binary -- that fallback is ACPI-only.",
           "default": "auto"
         }
       },