Repository navigation
Conversation
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
cquil11
marked this pull request as ready for review
October 6, 2026 15:58
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
cquil11
changed the title
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
cquil11
force-pushed
the
feat/docker-launcher
branch
from
October 12, 2026 16:37
b2b401e to
bd2b94d
Compare
|
|
…oncake-master service (NVIDIA#595) Connect TRT-LLM's Mooncake Store connector to the managed mooncake-master service for trtllm-serve and Dynamo workers. Render a client config per role, with shared service defaults and per-role overrides, and validate incompatible settings before launch. Document global_segment_size inheritance and per-rank sizing, provide both frontend examples, and refresh the Dynamo launch snapshot for the default OTel collector. Validation: make check (4,535 passed), 107 focused Mooncake/snapshot tests, both recipe dry-runs, and the strict documentation build. Signed-off-by: Yuewei Na <nv-yna@users.noreply.github.com>
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
…s it Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
feat(sa-bench): Add v1 chat completions
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
feat(sa-bench): cache generated random datasets across runs
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
feat(cli): bake dynamo and sa-bench deps into a container image
…e names Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
…kdown, prefer ACPI in auto (NVIDIA#481) * fix(power): DCGM CPU power is the CPU rail; publish the rail + SysIO breakdown, prefer ACPI in auto DCGM has no CPU power backend of its own. Its sysmon module reads the ACPI power_meter hwmon channels by power1_oem_info label, so each CPU-entity field is one ACPI rail (NVIDIA/DCGM modules/sysmon/DcgmSystemMonitor.cpp, DcgmModuleSysmon.cpp): 1130 DCGM_FI_DEV_CPU_POWER_WATTS <- "CPU Power Socket N" = cpu_rail 1132 DCGM_FI_DEV_SYSIO_POWER_UTIL_CURRENT <- "SysIO Power Socket N" = soc 1131 DCGM_FI_DEV_CPU_POWER_LIMIT_WATTS <- "Grace Power Socket N" cap only No field reports the Grace envelope's draw. DCGM-mode power_w (field 1130) is therefore the CPU rail, ~53 W/socket against a ~100 W ACPI envelope on the GB200 reference runs, and has been silently under-reporting CPU power relative to ACPI-mode runs. Rail vocabulary (cpu_rails): - DCGM_FIELD_RAIL_KINDS {1130: cpu_rail, 1132: soc}, DCGM_PRIMARY_FIELD_ID, field names and hwmon labels, in the one module that owns rail names. - cpu_sample.dcgm_rail_readings(): the one place field ids become rails. 1130 yields the dcgm primary (power_w, unchanged) and the cpu_rail component; 1132 yields soc. Host reader and scrape parser both call it. Producers: - Python host collector watches 1130+1132, fills cpu_rail_w/soc_w, and records a power_fields table (field, rail, hwmon label, column) in the node metadata. - Rust cpu-power-exporter watches both fields and publishes one cpu_power_dcgm_watts{socket,field_id} sample per field, 1130 first so a legacy collector still meets it first. `--source auto` now tries ACPI first and falls back to DCGM, matching the Python collector; DCGM-first is how run 430661 lost the socket envelope its nodes could provide. - cpu_parser reads the field_id label; an unlabelled sample (older exporter) is 1130, so old binaries keep working and now also get cpu_rail_w filled. Report: - power_energy_report: CpuSamples.sources; DCGM-sourced runs get a "CPU power source is DCGM field 1130 = CPU rail only ... not comparable with ACPI-mode runs" warning. Numbers are unchanged: report JSON for samples 430661 (DCGM) and 3054237 (ACPI) is identical to the pre-change output except for that warning. Docs: cpu-power-telemetry.md (field -> rail table, auto order, column semantics), config-reference.md, dcgm-4.7-runtime-support.md, schema docstring; stale "blank for DCGM" comments corrected. Tests: cpu_rails/cpu_sample/cpu_parser/cpu_power/collector/energy-report fixtures updated to the new contract plus new cases (labelled body, missing 1130, garbage field_id, rail-only warning on/off); Rust render_dcgm_metrics unit tests. * refactor(power): one DcgmPowerField record per field; note 1133 as unverified; review follow-ups Address the first review on NVIDIA#481: - cpu_rails: collapse DCGM_FIELD_NAMES / DCGM_FIELD_HWMON_LABELS / DCGM_FIELD_RAIL_KINDS into one DcgmPowerField(field_id, name, hwmon_label, kind) record per field (DCGM_POWER_FIELDS, DCGM_FIELD_BY_ID) so the three facts cannot drift; the derived dicts consumers use are built from it. Document why 1131 (envelope *cap*) and 1133 (Module Power, scope on GB200/GB300 unverified on live hardware) are deliberately excluded; the docs table now lists 1133 as unverified/excluded and gives the dcgmi + hwmon commands that would settle it. - cpu_parser: state that the new parser is order-independent and that the exporter's "1130 first per socket" emission order exists only for older srtctl parsers, so it is not removed as dead ordering later. - test_power_energy_report: explain why the rail-only warning test loads through the CSV (CpuSamples.sources comes from the `source` column; direct fixtures leave it empty). - test_cpu_power: a DCGM value for an entity we never enumerated is dropped (covers the guard Codecov flagged). - test_cpu_rails: pin the record table, 1130-first order, and the 1131/1133 exclusions. Report JSON for 430661 / 3054237 still identical to the pre-change output except the warning; 132 power tests, 14 cargo tests, ruff clean, ty count unchanged. * fix(postprocess): say why the power energy report was skipped when power samples exist `_build_power_energy_report` logged every PowerReportError at DEBUG, which is right when telemetry is off (no power CSVs, the routine case) but hides the real findings: a run that collected CPU/GPU power and still has no power_energy_report.json. Auditing the sample runs, two such gaps were only diagnosable by re-running the report offline: 3090377 profile_export_aiperf.json: not found (benchmark died, exit=1, before aiperf wrote its summary; no profiling window) 648928 cpu/nvl72d020-T08/socket0: window narrower than the sample spacing Now, when any samples.csv exists below the log dir, the skip is logged at INFO as "Power energy report skipped (power samples present): <reason>"; without power samples it stays at DEBUG. Post-processing still never fails the benchmark. Tests: the quiet path pins DEBUG; a new case builds the 3090377 shape (profiling records, no aiperf summary, a CPU CSV) and asserts the INFO line names the missing file. * fix(cpu-power-exporter): auto mode degrades instead of failing on dead ACPI or a partial DCGM field set Two regressions the review of NVIDIA#481 found in the new ACPI-first `auto`: 1. ACPI was trusted on discovery alone. A node that exposes power_meter channels which all read zero would be committed to ACPI and publish zeros for the whole run -- 0 J that looks like a measurement. Now every discovered sensor is read, and if none reports a finite positive value the probe is retried once after 1 s (hwmon averages can be 0 on the first poll after boot); only then does auto fall back to DCGM, at WARN with the sensor count. `--source acpi` bails on all-zero, matching its existing strictness. This cannot rescue a node via DCGM in practice -- DCGM's sysmon reads the same power1_average files, so it sees the same zero -- but it stops zero pollution and names the problem in the exporter log. 2. Watching [1130, 1132] in one field group made DCGM mode all-or-nothing. A libdcgm that rejects 1132 for CPU entities (older release, or a sysmon without SysIO) failed where 1130 alone had always worked, and under ACPI-first auto DCGM is the last resort, so that meant no CPU power at all. `DcgmReader::new` now tries [1130, 1132] then [1130]; the fallback is logged at WARN with DCGM's reason, `DcgmReader::fields()` reports what is actually watched, and read_watts sizes its buffer from that. Tests: probe dead when every sensor reads 0; live when any reads positive; retry rescues a sensor that goes 0 -> value between passes; field-set candidates all start with 1130 and end at [1130]. 18 cargo tests pass. * fix(cpu-power): host collector walks the same auto ladder as the exporter; document it The Python host collector (`srtctl.core.cpu_power`, the telemetry.cpu_power leg) had the two gaps the exporter fixed in the previous commit: - `create_reader("auto")` committed to ACPI on discovery alone. It now requires at least one sensor to read a positive value, retrying once after 1 s (`AcpiPowerMeterReader.probe_live`), and steps down to DCGM at WARN with the sensor count otherwise. `--source acpi` fails on dead sensors instead. - `DcgmCpuPowerReader` watched [1130, 1132] in one field group and failed outright if libdcgm refused the set. It now tries the candidates in `DCGM_FIELD_SET_CANDIDATES` ([1130, 1132], then [1130]), cleaning up the rejected field group, and logs at WARN when it lands on the rail alone. `power_field_ids` reports what is watched; `read_watts` accepts only those fields; `_sensor_kinds()` expects only those kinds; metadata gains `power_fields[].watched` so a per-node file shows which rung it landed on. The collector runs as a standalone srun process, so `main()` installs a stderr logging handler; the ladder's INFO/WARN lines land in the node's telemetry_cpu_power.<node>.out. Docs: cpu-power-telemetry.md spells out the four-rung ladder and what a missing reading looks like in the CSV -- a component rail that cannot be read is blank, never 0, for every cause (fallback to 1130-only, non-OK status, zero/non-finite); a socket missing its primary yields no row rather than a blank power_w; rails never substitute for the primary or join total_power_w. config-reference.md and the schema docstrings say the same in one line; schema-reference.md regenerated. Tests (8 new): DCGM fallback to 1130 when 1132 is refused (cleanup, WARN, no soc kind expected, metadata watched flags); fail when every set is refused; ACPI probe dead / live / retry rescues; auto steps dead ACPI -> DCGM with the count in the log; live ACPI never constructs DCGM; explicit acpi fails on dead sensors. 258 power/postprocess/docs tests pass. * fix(cpu-power-exporter): log every auto-mode fallback to DCGM at WARN docs/cpu-power-telemetry.md promises every step down the auto ladder is logged at WARN, but the 'no ACPI sensors found' and 'hwmon discovery error' fallbacks in init_metrics_state used info!, so a node landing on DCGM's half-size CPU rail number could do so without a warning in its .out file. The dead-sensor fallback was already WARN; the Python host collector already warns on every step. Logging change only. Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com> * docs(schema): CpuPowerExporterConfig.source description reaches schema-reference.md The source field's description was an attribute docstring (a bare string after the field), which schema_docs never reads -- it takes an Attributes: block in the class docstring or a # comment above the field -- so docs/schema-reference.md showed an empty description for cpu_power_exporter.source. Moved into an Attributes: block (adding port, which had none either) and regenerated with srtctl schema-docs. No behaviour change. Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com> * fix(cpu-power): 0 W is "sensor not reporting"; ACPI liveness requires a live socket total Every reader accepted 0 W as a sample (>= 0 in the Rust exporter's read_acpi_watts, the Python host collector's read_watts, and both scrape parser families), so a Grace envelope stuck at 0 produced a power_w=0 row and a total_power_w that summed it in -- integrating to 0 J and passing for a measurement -- while docs/cpu-power-telemetry.md promised zero is dropped. The startup probe compounded it: any positive sensor made ACPI "live", so a dead envelope beside a live CPU rail was accepted for the whole run. power1_average reports 0 only when the sensor has produced no reading (a live Grace socket draws tens of watts), so 0 is now filed as missing at every read site, and liveness means at least one socket-total channel reads positive -- that is the rail that becomes power_w. With 0 treated as missing, the existing pivot and aggregate rules do the rest: a socket with a stuck total gets no row, and the node total is blank rather than partial. A node with rails but no live total steps down to DCGM in auto (WARN) or exits under --source acpi; the Python ACPI reader already refused to construct without a total channel. Falling back to DCGM for a dead envelope was considered and rejected as the fix: DCGM's sysmon reads the same hwmon files, so it cannot recover the envelope -- it would only re-read the CPU rail via libdcgm and label it. Tests: zero total is missing / socket dropped / node total blank (host collector), probe ignores live rails with no live total and counts only totals (Python + Rust), zero total and zero 1130 in scrape bodies produce no row (parser), zero rail omitted from the exporter body (Rust). All fail on the previous source. Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com> * test(cpu-power-exporter): pin the auto ladder and the --source acpi contract init_metrics_state mixed the ACPI decision with DcgmReader::new(), which needs libdcgm, so nothing proved that auto is ACPI-first or that --source acpi errors instead of stepping down: the probe tests passed regardless of ladder order. The ACPI leg is now decide_acpi(mode, root, probe) -> Use | FallBack(reason) | Skip, with the liveness probe injected, and init_metrics_state only reaches DCGM on FallBack/Skip. While there, discovery now rejects a sensor set with no socket-total channel before probing, naming the domains it did find -- the Python collector already refused to construct in that case; the exporter used to run the 1 s probe retry and report "no total read positive" instead. Eight tests: auto uses live ACPI; auto falls back (with the reason) on no sensors / no total channel / dead total / unreadable root; acpi errors on each of those; acpi uses live sensors; dcgm never runs the probe. A mutation that makes acpi fall back like auto fails explicit_acpi_errors_instead_of_falling_back. 30 cargo tests pass. Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com> * docs(observability): cpu_power_exporter source ladder and DCGM power_w semantics The docs restructure on main (NVIDIA#565/NVIDIA#566) moved the telemetry.cpu_power_exporter prose from config-reference.md to observability.md. Carry this branch's two edits to it there: the auto ladder (live ACPI socket total, then DCGM 1130+1132, then 1130 alone) and the statement that DCGM-mode power_w is field 1130 = the CPU rail, about half the ACPI envelope, flagged by the energy report. * fix(cpu-power): DCGM liveness requires a positive field 1130; SysIO alone cannot vouch The DCGM twin of the ACPI socket-total rule. The exporter accepted DCGM when any watched field read positive, so a node where only SysIO (1132) reported would serve soc_w for the whole run with no power_w and no socket rows; the Python host collector had no liveness check at all and would accept a DCGM reader that returned nothing. Both now require at least one CPU entity to return a positive 1130 -- the value filed as power_w -- on a first read, and otherwise step down (auto, WARN) or exit (explicit dcgm) with the reason. Rust: dcgm::has_live_cpu_power() replaces the any-field check in init_metrics_state. Python: create_reader's dcgm factory is _create_dcgm_reader, which counts DCGM_KIND readings from the shared classifier -- the same test the row path applies -- and closes the handle on refusal. The row path was already correct (verified: a socket with only 1132 is dropped by pivot_socket_samples; a 1132-only scrape yields no sockets), so the change is to source selection only. Tests: Rust 1132-only / 1130-None / empty are not live, one live 1130 is; Python dcgm refused on 1132-only and on 1130 = 0, accepted on one live socket, auto fails with both reasons and two WARNs when ACPI is dead too. Dropping the field filter fails 2 cargo tests; reverting the factory fails 3 pytest cases. 33 cargo, 251 power/schema pytest pass. Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com> --------- Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com>
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
feat(sa-bench): isolate SA-Bench dependencies and sanitize baked image names
…r LISTEN (NVIDIA#607) Sidecar gRPC ports started at 50051, inside Linux's default ephemeral range (32768-60999), and the wrapper polled readiness with a /dev/tcp connect. A loopback connect can be handed the destination port as its source and connect to itself with nothing listening: the sidecar starts early, and the TIME_WAIT socket left on 127.0.0.1:<port> makes the engine's gRPC bind fail with EADDRINUSE for 60 s. Move the default base to 19000, between SGLang NCCL/TokenSpeed and vLLM's scan ranges, and wait for a LISTEN entry in /proc/net/tcp{,6} instead of connecting, which also covers sidecar_port overrides inside the ephemeral range. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(services): add file readiness probes * docs: fix file readiness example link * fix(services): address file readiness review feedback
…VIDIA#605) ty 0.0.87 adds invalid-mutable-override, which rejects narrowing the base classes' ClassVar[str | None] attributes to ClassVar[str] in AtomeshFrontend.required_backend and SGLangRouterFrontend.log_label. Match the base declarations, and pin ty so new rules arrive as a deliberate bump instead of breaking CI on unrelated PRs.
…emon to ask (NVIDIA#592) * fix(power): read the kernel sync flag when clock_sync_check has no daemon to ask Hosts without systemd/D-Bus or a chrony/ntp CLI (container-rooted Slurm nodes) failed the probe although the kernel reported a disciplined clock; adjtimex(2) now vouches as a last resort and the failure path keeps each probe's stderr. Signed-off-by: Wenyao Gao <wenyao.gao28@gmail.com> * refactor(power): spell out the adjtimex clock probe and name the captured probe outputs Signed-off-by: Wenyao Gao <wenyao.gao28@gmail.com> * fix: use kernel maxerror for clock synchronization checks Signed-off-by: Wenyao Gao <wenyao.gao28@gmail.com> * test: isolate clock probes from host time services Signed-off-by: Wenyao Gao <wenyao.gao28@gmail.com> --------- Signed-off-by: Wenyao Gao <wenyao.gao28@gmail.com>
* feat(trtllm): enable disaggregated OpenEngine sidecar workers Signed-off-by: tanmayv25 <tanmay2592@gmail.com> * docs(trtllm): link sidecar example from site guide Signed-off-by: tanmayv25 <tanmay2592@gmail.com> * feat(trtllm): pin OpenEngine sidecar example to nightlies Signed-off-by: tanmayv25 <tanmay2592@gmail.com> * fix(trtllm): colocate sidecar example and cover discovery Signed-off-by: tanmayv25 <tanmay2592@gmail.com> * feat(trtllm): validate GLM-5.2 sidecar recipe on GB300 Signed-off-by: tanmayv25 <tanmay2592@gmail.com> * docs(examples): simplify sidecar example description Signed-off-by: tanmayv25 <tanmay2592@gmail.com> * test(trtllm): reject sidecars with direct frontend Signed-off-by: tanmayv25 <tanmay2592@gmail.com> * fix(trtllm): pin HF snapshot across sidecar launch and benchmark Signed-off-by: tanmayv25 <tanmay2592@gmail.com> * fix(trtllm): scope snapshot pin to supported launch path Signed-off-by: tanmayv25 <tanmay2592@gmail.com> * fix(trtllm): preserve model ID in pinned sidecars Signed-off-by: tanmayv25 <tanmay2592@gmail.com> --------- Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
…s user-writable TRT-LLM endpoints that fit on one node now run every rank in one container under mpirun instead of being rejected. Containers ignore the image's USER and run with umask 0000, as pyxis remap-root does, so the host-side srtctl can keep writing next to their outputs. Signed-off-by: Cam Quilici <cjquilici@gmail.com>
* fix(power): prove worker GPU UUIDs before excusing idle GPUs on worker nodes The plan's CUDA index and the exporter's device index are joined by number alone, so idle GPUs an exclusive node's exporter reports are excused only when a bare-host probe (scontrol listpids joined to nvidia-smi compute apps) shows each worker holding the UUIDs sampled at its planned indices, and a disagreement blocks publication as worker_gpu_uuid_mismatch. Pool-node GPUs stay unexpected_device, every exporter row is kept as evidence, and node/role/total GPU energy counts worker GPUs only. Signed-off-by: Wenyao Gao <wenyao.gao28@gmail.com> * test: cover worker GPU probe parsing Signed-off-by: Wenyao Gao <wenyao.gao28@gmail.com> --------- Signed-off-by: Wenyao Gao <wenyao.gao28@gmail.com>
…NVIDIA#591) * feat(power): classify the superchip module power rail The module is a reference rail like the component rails: never the socket's power and never summed into the node total, because it contains the GPUs. * feat(power): persist module_w in CPU samples v3 and host CSV v5 Readers still accept schema 1 and 2. * feat(power): merge the NVML module power family into CPU scrapes A firmware-bound ACPI module meter wins over NVML for the same socket. * feat(power): export superchip module power per Grace socket from NVML Both GPUs of a superchip report the same module meter, so the socket sample is their mean. A host without libnvidia-ml.so logs one line and keeps serving the ACPI families. An ACPI Module Power Socket label classifies to the same module kind. * docs(power): document the NVML module rail and CPU samples v3 * refactor(power): document the metrics composition and mirror the ACPI label order Signed-off-by: Wenyao Gao <wenyao.gao28@gmail.com> * fix(power): read module power from NVML only The ACPI Module Power Socket meter is unbound on every firmware seen, so the precedence it had over NVML was never exercised. Drop the classification and the host collector's module_w column: NVML is the one module source and the one verified on hardware. Signed-off-by: Wenyao Gao <wenyao.gao28@gmail.com> * fix(power): poll NVML module power off the scrape path The module series is pre-rendered by its own thread, like the ACPI cache, so a stalled nvmlDeviceGetFieldValues cannot stall a scrape. A module socket the CPU family does not publish is reported once at startup, since the NUMA-node-as-socket mapping is verified on GB200 only. Signed-off-by: Wenyao Gao <wenyao.gao28@gmail.com> * fix(power): read the 1 s module average and drop sockets whose GPUs disagree Field 186 is no longer a fallback, so one series never mixes averaging windows. Both GPUs of a GB200 module report one meter (spread under 2% measured), so readings more than 5% apart are skipped with a warning. Signed-off-by: Wenyao Gao <wenyao.gao28@gmail.com> * docs(power): source the module meter definition and record the GB200 readings Signed-off-by: Wenyao Gao <wenyao.gao28@gmail.com> * fix(power): withhold a stale module snapshot and warn once per disagreeing socket A stalled NVML call used to freeze module_w at its last value; a snapshot older than 3 s is now left out so the gap is visible. The spread warning fired on every poll; it now fires when a socket starts disagreeing and when it recovers. Signed-off-by: Wenyao Gao <wenyao.gao28@gmail.com> * docs(power): quote the module source as written and record the NUMA-to-socket test The NVIDIA table describes the one-GPU CG1 module, so the two-GPU behaviour is stated as measured. The residual is reported without attribution, and the socket mapping now cites the CPU-pinned load test. Signed-off-by: Wenyao Gao <wenyao.gao28@gmail.com> * docs(power): drop the last references to an ACPI module meter The host collector stays at schema 4 without module_w and the Python exporter cannot classify a module rail, so the v5 bullet and the kind list described a path 60045c1 removed. Signed-off-by: Wenyao Gao <wenyao.gao28@gmail.com> * refactor(power): detect the wide CPU CSV by its component rail columns COMPONENT_RAIL_COLUMN_NAMES is the tuple the branch rederived from two header constants. Signed-off-by: Wenyao Gao <wenyao.gao28@gmail.com> * refactor(power): derive the frozen v2 CPU samples header from v1 The seven shared columns were spelled out three times; test_cpu_power_contract still pins the v2 literal. Signed-off-by: Wenyao Gao <wenyao.gao28@gmail.com> * refactor(power): name the socket check in the NVML startup failure The first read can also be empty because every socket failed the spread guard, so the message no longer blames the GPUs; the socket list is computed once. Signed-off-by: Wenyao Gao <wenyao.gao28@gmail.com> * chore(power): drop restating comments from NVML module power Signed-off-by: Wenyao Gao <wenyao.gao28@gmail.com> * fix: keep CPU telemetry available during NVML startup Signed-off-by: Wenyao Gao <wenyao.gao28@gmail.com> * refactor: clarify NVML polling names and comments --------- Signed-off-by: Wenyao Gao <wenyao.gao28@gmail.com>
SGLang binds each KV-event publisher at the configured port plus its DP rank (the attention-DP rank, or the replica rank without attention DP). Two DP workers on one node got consecutive base ports, so worker 0's rank 1 bound worker 1's base and failed with "Address already in use". NVIDIA#473 reserved a dp-size block, but only in Dynamo sidecar mode with KV events, and it read only dp-size. Reserve the block for every SGLang process, as the vLLM backend does for its local DP ranks, and size it with SGLangBackend.num_dp_ranks (dp-size x attn-dp-size). sgl-project/sglang#41818 added --attn-dp-size and left dp-size counting replicas; the deprecated dp-size + enable-dp-attention spelling resolves to the same layout. The sidecar follower-publisher check, the replica NCCL-port rule and the AIPerf follower scrape now read the same resolver, so --attn-dp-size works there too. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Cam Quilici <cjquilici@gmail.com> # Conflicts: # src/srtctl/cli/submit.py # src/srtctl/core/processes.py # src/srtctl/core/slurm.py # src/srtctl/mock.py # tests/test_dry_run.py
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #47 (single-node Docker launcher). This PR adds only the multi-node part:
DockerLauncherruns a job's processes across several Docker hosts.Design
docker_hostsinsrtslurm.yamllists the job's nodes and defaults to this machine. The orchestrator runs wheresrtctl applyruns and carvesdocker_hostsinto roles with the sameNodes.from_nodelista Slurm nodelist uses, so aggregated, multi-node aggregated and disaggregated recipes need no changes.validate()now checks the recipe's node count againstdocker_hosts; the single-node checks on dedicated frontend, benchmark or infra nodes and on service pools are removed, since with several hosts those are just more nodes.LaunchSpecruns one task on each node in itsnodelist. Each task is adocker run --gpus all --network host --ipc hostcontainer, started directly on this machine and throughssh -n -o BatchMode=yes <node>on any other node. A launch that targets several nodes (the exporters) starts one task per node in parallel, and%Nin the log path becomes the node name, as with srun.127.0.0.1. With several,node_ipruns the existingip_utils/get_node_ip.shlookup on the target node over ssh, sonetwork_interfacebehaves as it does under Slurm.docker kill --signal=...on its node, because killing the localsshordocker runclient does not stop the container. When the job script exits, any container labelledsrtctl.job=<job_id>that is still running is removed on every host.ssh -tt, so it gets SIGHUP when its ssh client is signalled; host steps are stopped by signalling their local process group. Without this, the remote process kept running after the job.outputs/, the srtctl checkout) must exist at the same path on every host, as on a Slurm cluster's shared filesystem.Validation
Run on
h200, on two nodes held withsalloc(worker-12 and worker-13). Each node had Docker 29.2.1 and NVIDIA H200 GPUs.srtctl applyran on worker-12 withlauncher: docker. Model: Qwen3-0.6B (bf16). Image:lmsysorg/sglang:v0.5.20. Benchmark: sa-bench at ISL/OSL 128/128, concurrencies 4 and 8. All three runs used tree51a61e2b, which is identical to this branch's tree after the merge of #47.docker_hosts--dist-init-addrset to worker-12's node IP; 40/40 and 80/80 requests completed; no containers or host exporters left on either nodeA failed run (the prefill worker exited on a startup flag) stopped the job and removed both nodes' containers.