Repository navigation
chore: sync NVIDIA upstream main through c5b33eac - #53
Merged
Merged
Conversation
…A#574) * fix: resolve dependencies when installing staged Dynamo wheels * fix: install exact staged Dynamo artifacts and clarify index overrides
* refactor: share WorkerMode across backends and topology * refactor: define WorkerMode alongside Backend
* ci: maintain source diff counts on ready pull requests * ci: remove source diff comments when no lines change
…A#568) * docs(vllm): document sidecar compatibility and merge timeline * docs(vllm): pin sidecar nightly compatibility cutoffs * docs: give native sidecars a dedicated guide
…IA#577) * refactor: remove component performance dashboard implementation Signed-off-by: Yuewei Na <nv-yna@users.noreply.github.com> * docs: remove remaining component dashboard references Signed-off-by: Yuewei Na <nv-yna@users.noreply.github.com> * style: apply pre-push formatting Signed-off-by: Yuewei Na <nv-yna@users.noreply.github.com> * fix: preserve raw host capture when removing dashboard Signed-off-by: Yuewei Na <nv-yna@users.noreply.github.com> --------- Signed-off-by: Yuewei Na <nv-yna@users.noreply.github.com> Co-authored-by: Yuewei Na <nv-yna@users.noreply.github.com>
…emetry (NVIDIA#459) * feat(power): allow any benchmark client placement with DCGM power telemetry Drop the `benchmark.client_placement: head` requirement from `_validate_dcgm_power`. The check stood in for a clock guarantee: sample timestamps (collector, orchestrator host) and window boundaries (benchmark child) are both `time.time()` and the coverage check compares them directly. Same-host placement is a sufficient condition for that, not a necessary one; NTP-synchronised nodes within one allocation agree to milliseconds against validator tolerances of 0.5-3 s. The rule was also already bypassed by `client_dedicated_node: true`, which reserves the bench node from the tail. This makes the NTP assumption explicit in the docstring and docs instead of enforcing placement. `CLOCK_SOURCE` is unchanged so existing windows and out-of-tree adapters keep validating. Tests: telemetry loads with client_placement head/first_decode/last_decode and with client_dedicated_node. Full suite: 1937 passed; the 5 failures (apply_mock, sa_bench http-reuse x3, fingerprint affinity) reproduce on unchanged upstream/main on macOS. * feat(power): gate DCGM power runs on NTP-synchronised node clocks Relaxing benchmark client placement (previous commit) makes the power artifacts depend on wall clocks agreeing across nodes. Turn that from an assumption into a pre-flight check: before any server starts, the orchestrator probes every node whose clock feeds an artifact (head/collector, bench client, infra, workers) on the bare host with timedatectl -> chronyc -> ntpq, all read-only and unprivileged on stock images. A node that cannot prove synchronisation fails the job under telemetry.required and warns otherwise. New `telemetry.clock_sync_check` (default true) opts out on clusters where none of those tools are reachable. Skipped for EVAL_ONLY and CPU-power-only runs, which write no DCGM windows. This checks the daemon's lock state, not measured skew; it catches the gross case (chrony down, misconfigured image) rather than sub-second drift. Tests: tests/test_clock_sync_check.py covers node set (incl. dedicated bench node, deduped), bare-host srun, required-vs-best-effort outcomes, hung probe, and every skip path. docs/schema-reference.md regenerated. * fix(power): mark artifacts unpublishable when the clock-sync probe fails on a best-effort run The pre-server clock check lived in the orchestrator and never reached the power session, so under telemetry.required: false a failed probe only logged a warning and the manifest could still finalize with publication_valid: true on clocks nobody had shown to agree. The orchestrator now keeps the failing hosts and hands them to PowerTelemetrySession.record_clock_sync_failures(), which records the new clock_sync_unverified reason and lists them in manifest.clock_sync_failures; _finalize_manifest folds that into publication_valid. The exit-code policy is unchanged: required still aborts before servers start, best-effort still exits with the benchmark's code, matching how sample_gap_exceeded behaves. clock_sync_unverified is classified runtime-only for the offline validator because the probe result cannot be reconstructed from disk. Tests: session-level (window coverage valid, publication false, exit 0, offline validator rejects the package) and orchestrator wiring (failures reach the session); mutation-checked by removing the publication gate. * feat(power): record which daemon vouched for each node's clock sync The probe previously exited 0 silently, leaving clock_sync_<node>.out empty on success and the sweep log with only an aggregate "all N nodes OK". For a measurement-validity gate that is thin evidence: nothing recorded whether timedatectl, chronyc, or ntpq proved synchronisation, or what offset the daemon reported. Each probe branch now echoes its proof before exiting -- the daemon name, plus chrony's Reference ID / System time / Last offset lines or ntpq's selected-peer row -- and the orchestrator logs "clock_sync_check: <node> OK (<evidence>)" per node, reading the first line of the .out file. Failure behaviour is unchanged. Tests: per-node OK logging with evidence extraction, the evidence parser's edge cases, and a TestProbeScript class that runs CLOCK_SYNC_SCRIPT under real bash with fake probe tools on PATH to pin quoting, here-strings, and branch order (three pass paths, three fail paths). * fix(power): make the offline validator honour recorded clock-sync failures The producer gates publication_valid on manifest.clock_sync_failures, but _check_stored_evidence did not include it in the recomputation. On an honest package (stored false, failures recorded) the validator failed for the wrong reason ("recomputed True"); with the stored verdict flipped to true the two agreed and the package passed outright. The recomputation now reads the persisted clock_sync_failures list through one helper shared with the wire check: absent means [] (packages predating the gate never ran the probe), malformed is reported and treated as unpublishable so a bad field cannot rescue the verdict. clock_sync_unverified stays runtime-only -- the probe is not reconstructible, but the hosts it flagged are on disk. Tests: the earlier session test now requires "stored publication_valid is false" and no "recomputed True"; added the tampered-verdict case, the honest case, a legacy manifest without the field, and three malformed shapes. Removing the new term fails all seven. Found by edwingao28 in review.
…IA#572) * feat(power): select the GPU power exporter through a profile table GPU power telemetry hard-coded dcgm-exporter in four places: the parser (DCGM_FI_DEV_POWER_USAGE and the gpu/UUID labels), the manifest wire values, the default launch command, and the tachometer filter for the shared endpoint. The launch side was already exporter-agnostic through the cluster default_gpu_exporter block, so a ROCm cluster could start an exporter but the collector could not read it. Move those four facts into one table row, PowerMetricProfile in srtctl.core.power.profile, keyed by a new power_profile field on the exporter block. The dcgm row reproduces the previous behaviour and stays the default. The amd-device-metrics row reads rocm/device-metrics-exporter (gpu_power_usage keyed by gpu_id, identified by serial_number, gpu_gfx_activity as the utilization rider, entrypoint script as the pyxis command). The parser, session, manifest and validator read the row instead of the DCGM constants; the manifest records power_profile alongside source_metric and power_scope so a retained bundle is self-describing, and the offline validator checks those keys against the named row (a manifest without the key is DCGM). A recipe that enables telemetry without naming any collector now inherits the cluster default_gpu_exporter, so one recipe measures power on NVIDIA and AMD clusters; recipes naming an exporter or only a CPU leg are unchanged. Signed-off-by: Cam Quilici <cjquilici@gmail.com> * Add AMD power exporter example and launch snapshot * refactor(power): default power_profile to dcgm and drop redundant guards Validate the profile name once in the schema, index POWER_PROFILES directly, remove the dataclass self-checks on the static table, and drop the telemetry_stage re-export of the DCGM command template. Signed-off-by: Cam Quilici <cjquilici@gmail.com> * refactor(power): configure exporter power metrics instead of naming a profile Replace the built-in profile table with a `power` block on the exporter config (metric, scope, index and identity labels, utilization riders, instance labels, tachometer handling). Unset still means DCGM, so NVIDIA configs are unchanged; any other exporter is described in YAML and needs an explicit command. The manifest keeps recording source_metric and power_scope, and the offline validator no longer pins them to a table. Signed-off-by: Cam Quilici <cjquilici@gmail.com> * chore: remove accidentally committed docs build output Signed-off-by: Cam Quilici <cjquilici@gmail.com> * refactor(power): split GPU exporter config into gpu_labels and gpu_metrics Replace the single power block with two optional blocks on the exporter: gpu_labels (index, identity, instance) and gpu_metrics (power, gpu_util, sm_active, one entry per recorded metric). Each defaults to DCGM when unset, so existing configs and manifests are unchanged. A non-DCGM power metric must name its labels and command. Tachometer handling becomes two optional top-level fields derived from the power metric when unset. Signed-off-by: Cam Quilici <cjquilici@gmail.com> * feat: retain GPU temperature in power telemetry Signed-off-by: Wenyao Gao <wenyao.gao28@gmail.com> * refactor: share the temperature value contract Signed-off-by: Wenyao Gao <wenyao.gao28@gmail.com> * feat(power): configure the GPU temperature metric through gpu_metrics Add gpu_metrics.temperature next to power, gpu_util and sm_active. DCGM keeps DCGM_FI_DEV_GPU_TEMP as the default; other exporters name their own metric (gpu_junction_temperature on AMD MI3xx) and are read by their configured labels. The manifest records temperature_metric. Signed-off-by: Cam Quilici <cjquilici@gmail.com> * fix(telemetry): scrape the tachometer GPU exporter with its power mapping With power telemetry off, the cluster default_gpu_exporter runs as the implied dcgm-exporter service, whose scrape filter, endpoint prefix and GPU metadata were hardcoded to DCGM's. Carry the exporter's mapping into the service options so a non-DCGM exporter keeps its own. Signed-off-by: Cam Quilici <cjquilici@gmail.com> * refactor(power): declare the GPU exporter kind instead of inferring DCGM Add kind: dcgm | custom to the exporter config. dcgm (the default) keeps the built-in command, DCGM labels and metrics and the dcgm tachometer scrape; custom must set command, gpu_labels and gpu_metrics and is scraped as served. The tachometer scrape is a table keyed by kind, so nothing compares a metric name to DCGM's, and the tachometer_filter / tachometer_gpu_metadata fields are gone. Signed-off-by: Cam Quilici <cjquilici@gmail.com> * docs(power): present the AMD exporter as an example of a custom exporter State the exporter's values as that exporter's, at the pinned version, rather than as facts srt-slurm relies on, list the requirements that hold for any custom exporter, and give the example a scope that names its source metric instead of a measurement boundary. Signed-off-by: Cam Quilici <cjquilici@gmail.com> * chore(power): cleanup from review - docs: replace the stale "profile's" wording left from the removed power_profile field - schema: build the DCGM label and metric defaults from the DCGM mapping row and the contract's metric constants instead of repeating them - samples: key _parse_row's row widths by the SAMPLES_SCHEMA_VERSION constants rather than literal versions - contract: bound temperature by a physical range (-273.15..200 C) instead of DCGM's blank sentinel, so no DCGM constant sits in the shared contract - config: inherit the cluster GPU exporter unless a CPU leg is actually enabled, so cpu_power: {enabled: false} no longer blocks inheritance - docs: note that an inherited command replaces the built-in 100 ms DCGM command Signed-off-by: Cam Quilici <cjquilici@gmail.com> * fix(power): require kind: custom for custom GPU labels - schema: reject gpu_labels under kind: dcgm, so an exporter block that omits kind: custom fails at load instead of being launched and scraped as DCGM; a DCGM exporter can still override gpu_metrics - docs: describe sm_active as DCGM's SM-active fraction and ask that only a metric with that meaning and 0-1 range be mapped to it Signed-off-by: Cam Quilici <cjquilici@gmail.com> * test(power): stub the hostname lookup in the AMD exporter launch test The test's readiness budget is 0.2 s, and a cold DNS lookup of the fake node name can take longer, so the first run on some machines failed. Signed-off-by: Cam Quilici <cjquilici@gmail.com> * fix(status): annotate context managers as Generator for ty 0.0.85 Signed-off-by: Cam Quilici <cjquilici@gmail.com> --------- Signed-off-by: Cam Quilici <cjquilici@gmail.com> Signed-off-by: Wenyao Gao <wenyao.gao28@gmail.com> Co-authored-by: Wenyao Gao <wenyao.gao28@gmail.com>
NVIDIA#582) * perf(dsight): spool captured metric points to disk during progressive builds * test(dsight): cover conflicts at the final metric timestamp * fix(status): annotate context managers as generators
) * feat(frontends): add the Shepherd Model Gateway (SMG) router Register frontend.type: smg, a StaticRouterFrontend that launches `smg launch` with the allocated worker URLs: --worker-urls for aggregated workers of any backend, --pd-disaggregation with --prefill URL BOOTSTRAP_PORT and --decode URL for prefill/decode. Readiness counts SMG's GET /workers registry, workers are probed before SMG starts, and the Prometheus listener is bound on SMG_METRICS_PORT and scraped by tachometer. Add docs/smg.md, a vLLM aggregated and an SGLang P/D example with their launch snapshots, and tests. Signed-off-by: Cam Quilici <cjquilici@gmail.com> * feat(smg): run the recipe setup script in the SMG container Share vLLM Router's setup-script preamble as setup_script_preamble in static_router.py and use it for SMG too, so a recipe can pip install smg into the model image instead of pulling a separate router image. Signed-off-by: Cam Quilici <cjquilici@gmail.com> * fix(vllm): serve one API for direct DP workers no router expands A direct vLLM worker behind a router that does not expand node-local DP pools (smg) is one vllm serve that balances its DP ranks itself: the leader node serves the API and the other nodes run headless ranks. The backend gave every node of such a DP endpoint an HTTP port, so SMG was handed the headless nodes' URLs too. Keep the standard topology (an HTTP port on the leader only) for every direct worker launch except routers that expand node-local DP, as frontend.type: vllm already did. Add examples/vllm/smg-dep16.yaml (two-node DP16 with EP behind SMG) and document why SMG's --dp-aware rank pinning is not used for vLLM HTTP workers. Signed-off-by: Cam Quilici <cjquilici@gmail.com> * feat(backends): ask every backend whether a mode serves gRPC Promote is_grpc_mode(mode) to BackendProtocol. SGLang keeps grpc-mode, vLLM answers roles.<role>.args.grpc (vllm serve --grpc, served by smg-grpc-servicer), and the HTTP-only engines answer False. StaticRouterFrontend.worker_scheme now picks grpc:// from that one hook for every static router, so sglang-router's override goes away and smg advertises gRPC workers the same way. Signed-off-by: Cam Quilici <cjquilici@gmail.com> * feat(frontends): probe gRPC workers on their port before the router starts The pre-start probe only polled HTTP /health, so gRPC workers were handed to the router before they were up. Wait for a gRPC worker's port to accept connections instead (vLLM's gRPC server binds it after the engine starts), within the same health-check budget. Signed-off-by: Cam Quilici <cjquilici@gmail.com> * fix(mock): stub the static router's gRPC port probe The mock orchestrator patches every readiness wait; the gRPC worker port probe added to StaticRouterFrontend.start_frontends was not, so a dry run of a gRPC recipe waited out the health-check budget. Signed-off-by: Cam Quilici <cjquilici@gmail.com> * docs(smg): run vLLM prefill/decode over gRPC with NIXL SMG learns a vLLM worker's KV connector only over gRPC (the servicer reports kv_transfer_config in GetServerInfo), so vLLM P/D behind SMG uses grpc: true workers with engine.connector: nixl; SMG tags the prefill leg for a remote decode and relays the NIXL handoff to decode. Add examples/vllm/smg-disagg-grpc.yaml, its snapshot, and docs. Signed-off-by: Cam Quilici <cjquilici@gmail.com> * docs(smg): note that vLLM gRPC workers serve no Prometheus metrics Signed-off-by: Cam Quilici <cjquilici@gmail.com> * feat(smg): route TRT-LLM workers through SMG trtllm-serve serves Prometheus text at /prometheus/metrics; its /metrics is JSON iteration stats. A static router in front of trtllm-serve workers advertised /metrics for them, so the tachometer targets and the AIPerf metrics URLs read the wrong route. Each backend now names its native server's Prometheus path (native_metrics_path), and frontends answer the worker path through worker_metrics_path(backend): Dynamo keeps /metrics, frontends whose workers are the engine's own server use the backend's path. metrics_path stays the router's own route. return_perf_metrics, which mounts the trtllm-serve route, now defaults on behind every direct-worker frontend, not only trtllm_serve. Adds examples/trtllm/smg-agg.yaml and its launch snapshot. Signed-off-by: Cam Quilici <cjquilici@gmail.com> * docs(smg): link the examples by URL for the strict docs build Signed-off-by: Cam Quilici <cjquilici@gmail.com> * refactor(smg): inherit required_backend from Frontend Signed-off-by: Cam Quilici <cjquilici@gmail.com> * refactor(telemetry): pass no worker metrics path for a services-only job Signed-off-by: Cam Quilici <cjquilici@gmail.com> * refactor(backends): rename native_metrics_path to prometheus_metrics_path Signed-off-by: Cam Quilici <cjquilici@gmail.com> * fix(smg): stop waiting for gRPC workers once a critical process exits Signed-off-by: Cam Quilici <cjquilici@gmail.com> * fix(vllm): allocate no prefill bootstrap port, which vLLM never binds vLLM hands the KV cache over its NIXL side channel, so a static router such as SMG was told about a bootstrap port nothing listened on. Signed-off-by: Cam Quilici <cjquilici@gmail.com> * fix(frontends): do not scrape Prometheus metrics from gRPC workers A gRPC-mode worker (vllm serve --grpc) starts no HTTP listener, so worker_metrics_path is now per mode and None for gRPC roles; Tachometer and the AIPerf server-metrics URLs skip them. Signed-off-by: Cam Quilici <cjquilici@gmail.com> * refactor(backends): type is_grpc_mode and worker metrics paths with WorkerMode Signed-off-by: Cam Quilici <cjquilici@gmail.com> * fix(smg): reject prefill/decode layouts SMG cannot disaggregate Like the other direct-worker frontends, check the recipe layout at load: a P/D topology must use one engine in both roles, may not be TRT-LLM, and with vLLM must serve gRPC in both roles, since over HTTP the decode worker recomputes the prompt with no KV handoff. Signed-off-by: Cam Quilici <cjquilici@gmail.com> * fix(sglang): scrape gRPC-mode workers on their HTTP sidecar SGLang --grpc-mode serves /metrics and the profiler routes on an HTTP sidecar rather than the gRPC port, so dropping gRPC workers' metrics also dropped them for sglang-router and SMG jobs with SGLang gRPC workers. The allocator now gives each gRPC-mode SGLang leader a sidecar port, passed as --grpc-http-sidecar-port, and the static routers scrape and send profiler control there. vLLM's gRPC server has no HTTP listener and still gets no worker target. Signed-off-by: Cam Quilici <cjquilici@gmail.com> --------- Signed-off-by: Cam Quilici <cjquilici@gmail.com>
* fix(sglang): relay multinode DP sidecar KV events Signed-off-by: jain-ria <riajain@NVIDIA.com> * fix(sglang): delegate KV relay readiness to Dynamo Signed-off-by: jain-ria <riajain@NVIDIA.com> * fix(sglang): support automatic sidecar mode and disagg DP Signed-off-by: jain-ria <riajain@NVIDIA.com> * fix(sglang): allocate separate NCCL ports for regular DP Let native SGLang allocate per-replica NCCL ports for sidecar regular DP instead of reusing a fixed port across independent TP groups. Preserve fixed ports for TP and attention DP, and cover aggregate, prefill, and decode launch commands. Signed-off-by: jain-ria <riajain@NVIDIA.com> * docs: clarify SGLang sidecar section Signed-off-by: jain-ria <riajain@NVIDIA.com> --------- Signed-off-by: jain-ria <riajain@NVIDIA.com>
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merges NVIDIA/srt-slurm main through
c5b33eacinto the fork, including the upstream SMG frontend (NVIDIA#569). No conflicts.