Repository navigation
chore: sync NVIDIA upstream main through 3554ffa7 - #44
Merged
Merged
Conversation
… in place `roles.<role>.restart` (never | on-failure | always, with max_restarts and an exponential backoff) hands the role's workers to a WorkerSupervisor that ticks inside the process monitor: an exited unit is relaunched on the same node, GPUs and ports with a `_r<n>` step name, its crash log rotated aside, and the event recorded in worker_restarts.json and the lockfile. The unit is one engine of a worker, so a vLLM failover worker relaunches only the engine that died. A relaunched unit gets the health gate's budget to come up again; exhausted restarts hand the unit back to the role's `critical` flag. Rebased onto main (098e15a) as one commit (was 19deaa4, fa4731e, 0128a92, 6144980, 16b71ff) with these adaptations: relaunch_endpoint asks the role's own engine for its launch strategy (roles may run different engines); worker_ready_probe goes through the frontend registry (frontend_name_branches design rule); _start_workers groups only the role's processes; the worker-restart example has a launch snapshot; test stubs carry SrunConfig.kill_on_bad_exit. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
main's check_failures catches an engine that dies behind a launcher whose srun step stays up by scanning fatal_log_patterns; supervised processes skip that scan, so under a restart policy such a worker was never relaunched and never failed the job. The supervisor now runs the same scan on running steps, stops the step, and takes the normal backoff and relaunch path; a marker counts as a failure whatever exit code the stopped step returns. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…cache service examples/features/sglang-weight-cache.yaml runs SGLang's weight cache daemon (https://www.lmsys.org/blog/2026-08-21-sglang-fast-recovery/) as a generic `placement.per: worker` service: the daemon keeps the model's tensors in HBM and hands them to the engines over CUDA IPC, the engines start with `--weight-cache-mode client` and skip the disk load, and `roles.<role>.restart` relaunches a killed engine in place, where it maps the same tensors again. Socket and ready paths go through SGLANG_WEIGHT_CACHE_*_TEMPLATE under /dev/shm because SGLang's default /tmp is a per-container tmpfs under enroot. docs/sglang-weight-cache.md covers what runs, what srtctl owns versus what the recipe sets, how to test a relaunch, the sa-b200 numbers, and the limits (SGLang v0.5.19+, IPC quantization allowlist, no speculative decoding, the daemon argv written by hand, multi-node placeholders). No srtctl code change. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
reconcile() runs inside the process-monitor tick under the supervisor lock, and await_stop() on a sibling or on a step that printed a fatal marker held that tick for up to terminate_timeout per step, so check_failures() did not run for any other process meanwhile. The supervisor now sends SIGTERM on one tick and settles the stop on later ones through ManagedProcess.advance_stop (SIGTERM srun, then SIGKILL, each past its own deadline); the relaunch waits until every old step of the endpoint has exited. A fatal marker that arrives after max_restarts is recorded as the registry's failure (ProcessRegistry.record_failure): the supervisor consumed the log line and the step is still up, so check_failures would never see it otherwise. Signed-off-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
将已解析的配方传入提交前检查，使启用 CPU 功耗采集的作业在 exporter 缺失、不可执行或架构不匹配时于 sbatch 前失败。 Signed-off-by: Wenyao Gao <wenyao.gao28@gmail.com>
* feat: add worker_srun_options for inference worker steps Signed-off-by: Cam Quilici <cjquilici@gmail.com> * refactor: scope worker srun options to each role * docs: demonstrate role overrides of global srun options --------- Signed-off-by: Cam Quilici <cjquilici@gmail.com>
Require per-run performance reports with latency, throughput, agentic benchmark metrics and goal-linked diagnostic evidence. Keep concise agent instructions and a compact guide in the documentation. Signed-off-by: Yuewei Na <nv-yna@users.noreply.github.com>
Add compact guidance for short parallel debugging jobs and full-run allocation sizing from comparable history, with a two-hour fallback for a 60-minute measured run when no comparable history exists. Link the skill from the DSight guide and agent instructions. Signed-off-by: Yuewei Na <nv-yna@users.noreply.github.com>
Signed-off-by: Yuewei Na <nv-yna@users.noreply.github.com> Co-authored-by: Yuewei Na <nv-yna@users.noreply.github.com>
* refactor: replace protocols with abstract base classes * refactor: remove legacy protocol name aliases
Keep restart configuration on RoleConfig and preserve current role, engine, migration, and launch behavior. Add schema, dry-run, and mock-orchestrator regression coverage. Co-Authored-By: ishan <idhanani@nvidia.com>
Co-Authored-By: ishan <idhanani@nvidia.com>
feat(roles): per-role restart policy that relaunches an exited worker in place
Co-Authored-By: ishan <idhanani@nvidia.com>
Co-Authored-By: ishan <idhanani@nvidia.com>
…Schema, and MCP (NVIDIA#565) * docs: make schema metadata the single field reference for docs, JSON Schema, and MCP Co-Authored-By: ishan <idhanani@nvidia.com> * test: follow the Protocol -> Backend rename from main Co-Authored-By: ishan <idhanani@nvidia.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…N Schema, Mermaid diagrams, strict build on PRs (NVIDIA#566) * docs: make schema metadata the single field reference for docs, JSON Schema, and MCP Co-Authored-By: ishan <idhanani@nvidia.com> * test: follow the Protocol -> Backend rename from main Co-Authored-By: ishan <idhanani@nvidia.com> * docs: split the recipe guide into topic pages, task-based nav, strict site build on PRs - Split config-reference.md into cluster-config, engines, topology, frontends, benchmarks, observability and runtime-env pages; config-reference.md is now the recipe guide index. - Reorganize mkdocs.yml nav by task; add the four orphaned pages; exclude AGENTS.md and design/. - Fix broken links (repo files now link to GitHub) and anchors; CI runs mkdocs build --strict. - srtctl schema-docs also writes docs/schema/{recipe,cluster}.schema.json and docs/cli-reference.md (rendered from the new side-effect-free build_parser()); cli.md drops its copied flag tables. - MkDocs hook writes llms.txt from the nav on every build. - tests/test_docs_yaml.py checks every docs YAML block against the JSON Schema; fixes the stale telemetry.provider example and JSON Schema shorthands for roles.*.restart and services[].metrics. - MCP get_config_reference searches the topic pages; agent docs point at them. - README.md is the GitHub landing page; docs/README.md is the site home. Co-Authored-By: ishan <idhanani@nvidia.com> * docs: draw every Markdown diagram in Mermaid; render it in MkDocs and test against hand-drawn arrows Co-Authored-By: ishan <idhanani@nvidia.com> * chore: add SPDX headers to the new docs generator and MkDocs hook Co-Authored-By: ishan <idhanani@nvidia.com> * chore: rerun CI against main Co-Authored-By: ishan <idhanani@nvidia.com> * fix(docs): render REMAINDER positionals the same on Python 3.10 and 3.12 Co-Authored-By: ishan <idhanani@nvidia.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: ishan <idhanani@nvidia.com>
Co-Authored-By: ishan <idhanani@nvidia.com>
docs(examples): SGLang fast engine recovery with a per-worker weight-cache service
Keep the fork's record_launch_plan docs on upstream's split config pages and regenerate the schema reference. Signed-off-by: Cam Quilici <cjquilici@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merges NVIDIA/srt-slurm main through
3554ffa7into the fork.Conflicts:
core/schema.py: kept the fork'sClusterConfig.record_launch_plannext to upstream's new field comments, and added a field comment forOutputConfig.record_launch_plan.docs/config-reference.md: took upstream's split recipe guide and moved the fork'srecord_launch_plandocs to itsoutputsection and todocs/cluster-config.md.docs/schema-reference.md: regenerated.Note
Low Risk
Mostly documentation, CI, and agent guidance; conflict resolution preserves existing
record_launch_planbehavior with relocated prose only.Overview
Syncs the fork with NVIDIA/srt-slurm through
3554ffa7, resolving merge conflicts while keeping fork-specificrecord_launch_plandocumentation in the new doc layout (outputin the Recipe Guide and Cluster Config).Upstream brings a documentation restructure: the monolithic config reference splits into topic pages (benchmarks, engines, frontends, observability, cluster-config, etc.), ASCII architecture diagrams become Mermaid, and generated
docs/cli-reference.mdplus expandedsrtctl schema-docschecks are enforced in CI. A newdocsCI job runsmkdocs build --strict, andmake docsmirrors that locally.Agent and contributor guides now require
<run_dir>/perf-analysis.mdfor every Slurm run analyzed, point DSight skills to.agents/skills/, and mandate slurm-job-sizing before job submission. Design guidance shifts from Protocol interfaces toBackend/FrontendABCs with shared defaults—aligned with the codebase and reflected acrossCLAUDE.md, skills, anddocs/architecture.md.Reviewed by Cursor Bugbot for commit 3c0b8d4. Bugbot is set up for automated code reviews on this repo. Configure here.