[FEAT] Support fast engine recovery through weight cache

[QiuMike]

[RFC] Fast Recovery Framework for SGLang

Author: Michael Qiu qiudayu.qdy@antgroup.com; Siyu Liu @liusy58

Motivation

SGLang engine restarts are expensive. When an engine crashes, rolls out, or needs to restart for any reason, it must reload model weights from storage into GPU memory, re-initialize the distributed backend, recapture CUDA graphs, and re-warm JIT kernels — a process that takes 3–6+ minutes for large models (e.g., Qwen3-235B FP8 takes ~6.5 minutes on 4×GPU). In production serving environments, this translates directly to:

• P99 tail latency spikes: During restart, all in-flight requests fail or queue indefinitely.
• Reduced availability: Multi-minute recovery windows violate SLA targets for mission-critical deployments.
• Operational friction: Rolling updates, config changes, and failure recovery are all bottlenecked by the restart cycle.

Beyond crash recovery, the Weight Cache Daemon unlocks a broader set of production scenarios that are impractical today:

• Multi-instance weight sharing: A single daemon per GPU holds weights in memory; multiple engine instances (e.g., different DP ranks or independent services) map to the same IPC handles via zero-copy. Weights are loaded from disk and quantized exactly once per GPU, regardless of how many instances consume them. This eliminates redundant disk I/O and repeated post-quantization transforms.
• Priority co-serving: Run a high-priority online service and a low-priority batch job on the same GPU, backed by the same weight cache daemon. The low-priority instance can be evicted and re-spawned in sub-second time without reloading weights from disk, enabling flexible GPU time-sharing without the usual startup penalty.
• Active-standby failover: Deploy a standby engine alongside the primary, both backed by the same weight cache daemon. The standby maps weights via zero-copy and stays warm. When the primary fails, the standby takes over in < 1 second — no weight loading, no disk I/O. This achieves near-zero-downtime failover without dedicating a full set of GPUs to an idle replica, avoiding the expensive GPU resource waste of traditional hot-standby deployments.

This RFC proposes a Fast Recovery Framework with the goal of reducing engine restart time to < 10 seconds (cold restart) and < 1 second (warm standby switch). The framework addresses each startup phase independently, with the weight cache daemon as the first and highest-impact optimization.

Startup Time Breakdown

We profiled a complete SGLang engine startup for Qwen3-235B FP8 on 4×GPU (TP=4):

Phase	Time (s)	Time Range	Percentage	Notes
Server init & Tokenizer	~17.3	22:48:46 → 22:49:03	~4.4%	ServerArgs parsing, tokenizer loading
Init torch distributed	~4.7	22:49:03 → 22:49:08	~1.2%	NCCL init, 4-card
Load weight (disk)	~306–327	22:49:10 → 22:54:38	~79%	Disk I/O bound; slowest TP2=327s
KV Cache allocation	~0.5	22:54:38 → 22:54:39	~0.1%	194510 tokens
DeepGEMM JIT warmup (round 1)	~12.7	22:54:42 → 22:54:55	~3.3%	GEMM_NT_F8F8BF16 N=2304 K=4096
DeepGEMM JIT warmup (round 2)	~10.4	22:54:55 → 22:55:05	~2.7%	GEMM_NT_F8F8BF16 N=4096 K=2048
Capture CUDA graph	~34.9	22:54:38 → 22:55:13	~8.9%	Includes DeepGEMM warmup, 12 batch sizes
Server ready	~3.4	22:55:13 → 22:55:17	~0.9%	Tree cache init, warmup requests
Total	~390 (~6.5min)	22:48:46 → 22:55:17

Key insight: Weight loading from disk dominates at ~79% of total startup time. Even though CUDA graph capture + kernel warmup is the second largest cost at ~15%, it's dwarfed by the 5+ minutes spent on disk I/O. Eliminating weight loading is the single highest-leverage optimization.

---

Fast Recovery Framework: Multi-Phase Roadmap

The target is < 10s restart time. This requires optimizing every major startup phase:

Phase	Current (s)	Target (s)	Approach	Status
Server init & Tokenizer	~17.3	< 3	Lazy tokenizer init, config caching	Planned
Init torch distributed	~4.7	< 2	NCCL session reuse, persistent process groups	Planned
Load weight	~306–327	< 1	Weight Cache Daemon (CUDA IPC)	This PR
KV Cache allocation	~0.5	< 0.5	Already fast; pre-alloc from daemon	Planned
DeepGEMM JIT warmup	~23.1	< 0	Kernel cache persistence, parallel warmup	Planned
Capture CUDA graph	~34.9	< 3	CUDA graph serialization + replay	Planned
Server ready	~3.4	< 1	Skip warmup requests on restart	Planned
Total (single-node)	~390	< 10

Phase 1: Weight Cache Daemon (This PR)

Eliminates the dominant bottleneck (~79% of startup). A persistent GPU process holds post-quantized, TP-sharded weights and serves them to new engine instances via CUDA IPC zero-copy mapping. Weight loading drops from ~306s to <1s.

Current implementation supports single-node, TP-only deployments. Each GPU runs one daemon for its TP rank, communicating via Unix domain sockets and CUDA IPC (same-node GPU memory sharing).

Phase 2: CUDA Graph Serialization

Serialize captured CUDA graphs to shared memory or disk. On restart, replay the serialized graphs instead of recapturing. Eliminates the second-largest bottleneck (~9% of startup). Requires pool offset translation for address relocation.

Phase 3: Kernel Warmup Optimization

Persist DeepGEMM JIT-compiled kernels across restarts. Parallelize warmup across batch sizes. Reduces warmup from ~23s to <2s.

Phase 4: Server & Distributed Init Optimization

Lazy tokenizer initialization, NCCL session reuse, config caching. Reduces server init + distributed init from ~22s to <5s.

Phase 5: Multi-Node & DP/EP Support

Extend the Weight Cache Daemon to support multi-node (e.g., TP=16 across 2 machines) and DP/EP deployments. CUDA IPC remains the same-node GPU memory sharing mechanism — this phase adds primary-secondary daemon coordination across nodes.

Phase 6: Enhance robustness

The daemon is simple without any heavy logic, so itself is very robust, but if it been exited by other reasons, the client will still hold the GPU Mem, so if we restart the daemon, we could restore the GPU memory from client ref

• Primary-secondary daemon architecture: For multi-node TP (e.g., TP=16 = 2 nodes × 8 GPUs), one node runs the primary daemon that orchestrates the full distributed loading process, while the other nodes run secondary daemons. The primary coordinates daemon startup, distributed init (NCCL process group formation across nodes), and readiness signaling. Secondaries load their local TP shards in parallel with the primary, then serve IPC handles to engines on their own node via CUDA IPC (same as single-node). This preserves the fast IPC path per-node while adding cross-node coordination.
• Cross-node daemon lifecycle: Primary daemon manages the overall daemon cluster state — tracking which secondaries are alive, triggering coordinated restarts, and handling failover. If a secondary daemon dies, the primary can signal engines on that node to fall back to disk loading. If the primary dies, a secondary can be promoted.
• DP support: SGLang's data parallelism runs multiple engine instances (DP ranks) that share the same model weights. Without weight cache, each DP rank independently loads the full model from disk, wasting both load time and storage bandwidth. With the daemon, each GPU runs one daemon holding its TP shard; all DP ranks on the same GPU map to the same daemon's IPC handles (zero-copy), so weights are loaded from disk exactly once per GPU regardless of DP degree. This is primarily about supporting SGLang's DP deployment model — the benefit is eliminating redundant disk reads and weight quantization across DP ranks, not cross-GPU IPC sharing.
• EP support: Expert-parallel models have per-rank expert shards that differ across EP ranks. The daemon needs to cache per-EP-rank weight shards (keyed by ep_rank in CacheConfig), and support partial expert weight loading when only a subset of experts is assigned to a rank.
• CacheConfig extension: Add ep_size, ep_rank, and node_rank fields to CacheConfig for correct shard identification in EP and multi-node scenarios.
• Daemon discovery: Replace static Unix socket paths with a lightweight service discovery mechanism (e.g., etcd-based or file-based) so that engines can locate daemons across nodes without manual --weight-cache-socket configuration.

---

Phase 1: Weight Cache Daemon

Architecture

┌─ GPU i ──────────────────────────────────────────────────┐
│                                                          │
│  ┌───────────────────┐   cudaIpcMemHandle   ┌─────────┐ │
│  │ Weight Cache      │─────────────────────►│ Engine  │ │
│  │ Daemon (rank i)   │   (zero-copy)        │ Rank i  │ │
│  │                   │                      │         │ │
│  │ Holds:            │                      │         │ │
│  │ - TP-sharded      │                      │         │ │
│  │   weights (fp8)   │                      │         │ │
│  │ - weight_scale    │                      │         │ │
│  │ - workspace       │                      │         │ │
│  │ - all post-quant  │                      │         │ │
│  │   params/buffers  │                      │         │ │
│  └───────────────────┘                      └─────────┘ │
│                                                          │
└──────────────────────────────────────────────────────────┘

Coordination: Unix Socket /tmp/sglang_weight_cache_gpu{i}.sock

Each GPU runs one daemon process holding only its own TP rank's shard. The daemon caches the complete post-quantization state (after process_weights_after_loading()), including new parameters like weight_scale, workspace, repacked weights, etc.

Zero-Copy IPC Loading

The engine initializes the model on the meta device (no GPU/CPU memory allocation), then maps each parameter's data pointer directly to the IPC-mapped GPU tensor. The engine and daemon share the same physical GPU memory via CUDA IPC — no copy is needed.

Mode	Flow	Weight Load Time	GPU Memory	Use Case
daemon	Engine launches daemon → daemon loads from disk → engine maps IPC	< 0.1s (after daemon ready)	1× (shared)	First start; engine manages daemon lifecycle
client	Connect to pre-running daemon → map IPC	< 0.1s	1× (shared)	Engine restart; daemon pre-running
off	Normal disk loading	306–327s (235B FP8)	1×	Default; no cache

Process Lifecycle

First Start (--weight-cache-mode daemon):
  Engine → launch daemons (rank 0..N) → daemons load from disk (~5min for 235B)
         → engine loads from daemon via IPC (<1s)
         → engine runs normally

Engine Crash/Restart (daemon still alive, --weight-cache-mode client):
  Engine → connect to daemon socket
         → validate CacheConfig → match → IPC load (<1s)
         → mismatch → fallback to disk load (~5min)

Daemon Restart:
  Daemon → reload from disk → re-export IPC handles
         → subsequent engine restarts can use cache

Config Validation (CacheConfig)

Any mismatch in these fields triggers a full disk reload, ensuring correctness:

• model_path + model_arch — different model
• tp_size + tp_rank — different TP sharding
• dp_size — different DP strategy
• quant_method + quant_config_hash — different quantization
• dtype — different precision

---

Performance Results

Weight Loading: Disk vs IPC Zero-Copy

Model	Weight Size	Disk Load (s)	IPC Zero-copy (s)	Speedup
7B FP16	~14 GB	~30	<1	~3000×
70B FP16	~140 GB	~180–300	<1	~1800–3000×
235B FP8	~235 GB	~306–327	<1	~3000×

GPU-internal copy bandwidth: ~500–900 GB/s (H20 HBM). IPC handle mapping: ~10k handles/ms.

IPC Loading Path Optimization

During development, the IPC loading path was profiled and optimized:

Step	Before (s)	After (s)	Optimization
Meta device model init	0.3	0.3	—
IPC handle deserialization + param mapping	4.7	0.95	Pre-build param/buffer dicts for O(1) lookup instead of iterating model.named_parameters() per entry
Non-persistent buffer materialization	0.05	0.05	—
Total IPC load	5.05	1.30

End-to-End Startup: Qwen3-235B FP8 (4×GPU, client mode)

Test: disk load vs Weight Cache Daemon client：

Phase	Disk Load (s)	IPC Client (s)	Savings
Server init & Tokenizer	~17.3	~13.3	—
Init torch distributed	~4.7	~5.5	—
Load weight	~306–327	~0.63	~305s
KV Cache allocation	~0.5	~0.3	—
DeepGEMM JIT warmup	~23.1	~22.2	—
Capture CUDA graph	~34.9	~33.9	—
Server ready	~3.4	~4.4	—
Total	~390 (6.5min)	~80 (1.3min)	~310s (5.2min)

---

Implementation

Maintainability and Safety

This feature is relatively independent and has minimal invasiveness to the overall engine. Additionally, if the daemon process crashes or exits, it will not affect client-side inference. Since GPU memory is not released upon a crash (thanks to CUDA's reference counting), memory is only freed when both the daemon and the client processes exit. Therefore, this approach is very safe.

File Layout

python/sglang/srt/weight_cache/
├── __init__.py          # Package init, exports main classes
├── daemon.py            # WeightCacheDaemon process + CLI launcher
├── ipc_loader.py        # IpcModelLoader (BaseModelLoader subclass)
└── protocol.py          # CacheConfig, socket protocol, serialization helpers

Integration Points

Component	Change
LoadFormat.IPC_CACHE	New load format enum value in configs/load_config.py
--weight-cache-mode	Server arg: off | daemon | client in server_args.py
--weight-cache-socket	Server arg: path to daemon socket (for client mode)
IpcModelLoader	New BaseModelLoader subclass in weight_cache/ipc_loader.py
ModelRunner.load_model()	Dispatches to IpcModelLoader when cache mode is set
Engine._launch_weight_cache_daemons()	Launches daemon processes in daemon mode
ModelRunner (memory saver)	Disables CPU weight backup in IPC zero-copy mode

CLI Usage

# Standalone daemon launch (one command for all TP ranks):
python -m sglang.srt.weight_cache.daemon \
    --model-path /path/to/model --tp-size 8 \
    --load-format auto --dtype auto --quantization fp8

# Optinal:Single-rank daemon (advanced without Standalone daemon) x8 --gpu-id --tp-rank:
python -m sglang.srt.weight_cache.daemon \
    --model-path /path/to/model \
    --gpu-id 0 --tp-size 8 --tp-rank 0 \
    --dist-init-method tcp://127.0.0.1:29500

# Engine Client — connect to pre-running daemons (restart)
python -m sglang.launch_server \
    --model-path /path/to/model --tp-size 8 \
    --weight-cache-mode client

---

PR Breakdown (Phase 1)

#	Category	Description	Status
1	Core	Weight Cache Daemon + IPC Loader + Protocol + CLI args	Done
2	Core	Zero-copy IPC loading with meta device init	Done
3	Core	CacheConfig fingerprint validation	Done
4	Core	Single-command daemon launcher (all TP ranks)	Done
5	Core	IPC loading optimization (4.7s → 0.95s param mapping)	Done
6	Core	Memory saver integration (disable CPU backup in IPC mode)	Done
7	Test	Unit tests (CacheConfig, protocol, dispatch, cross-process IPC)	Done
8	Test	E2E test with real model loading	Done
9	Doc	Design document	Done

---

CI States

Latest PR Test (Base): ✅ Run #30099779370
Latest PR Test (Extra): ✅ Run #30327534784

[gemini-code-assist]

Code Review

This pull request introduces a persistent Weight Cache Daemon and a CUDA Graph serialization mechanism to enable fast engine restarts by caching post-quantized weights and CUDA graph metadata. The feedback highlights a few critical issues and optimizations, including a potential NameError in input_buffers.py due to a missing import, a native resource leak in cuda_graph_serializer.py if graph reconstruction fails, a misleading error message in engine.py during daemon startup failures, and a performance bottleneck in cuda_graph_inspector.py caused by excessive driver API calls on scalar arguments.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[liusy58]

/rerun-failed-ci

[alexnails]

two small comments and please go over:

• the actual comments in the code, some of them can be RM'ed or simplified
• the try except usage, some of it is necessary given the PR context, but some of it is unnecessary

[alexnails]

in next follow up: we can leave stale .sock/.ready files in edge cases

[galletas1712]

Hi @QiuMike @liusy58, this is great to see! We have a similar proposal in another RFC here: #27310 which would utilize something similar we have built in Dynamo (see https://github.com/ai-dynamo/dynamo/tree/main/lib/gpu_memory_service). So it seems like we are converging on a design.

A few questions:

In our design, we instead have the engine do all of the weight loading. Depending on whether the daemon already has weights or not, it will either exercise the regular model loading path (if it doesn't), or use the meta model initialization + IPC import (if it does).

We actually had an old prototype of the same architecture as this implementation. However, the weight daemon is actively holding a (CUDA) context because it does the loading of the weights - but there can be some classes of errors where holding onto a context causes the process to be poisoned, which means it needs to be killed (e.g. NVLink failure). If the daemon simply gives out CUDA allocations, no CUDA context is actually necessary allowing the daemon to resilient to more classes of failures. Another benefit is you don't need to deal with "fake sharding" in the weight daemon so it's less complex.

We've implemented an early version of active standby failover already within Dynamo, and one of the gaps here I see is how you would go about pause/resume with this architecture. Ultimately the secondary (we call this the "shadow" in Dynamo) would need to consume as few GPU resources as possible, so we would need to make sure the KV cache is unmapped, and free as many buffers as possible, while sharing the same set of weights. I'm wondering whether you have any thoughts on your approach to this problem with this architecture. One way to handle this is via a custom TMS backend: fzyzcjy/torch_memory_saver#79

The meta-model instantiation path is quite brittle. I see here that there are a few things that need to be hard-coded (some tensors are views of others and need to be rewired to the same storage, etc) that are in _rebuild_stale_views, among other issues (like some required tensors being tensor attributes but not registered as parameters/buffers etc). We are also running into similar problems, and a lot of the issues on this front are caused by replaying the host-side initialization. Is the plan to patch all of these instances one by one? Or should we have an integration test that allows "importing" from the daemon to always work?

[liusy58]

/rerun-failed-ci

[liusy58]

Hi @QiuMike @liusy58, this is great to see! We have a similar proposal in another RFC here: #27310 which would utilize something similar we have built in Dynamo (see https://github.com/ai-dynamo/dynamo/tree/main/lib/gpu_memory_service). So it seems like we are converging on a design.

A few questions:

In our design, we instead have the engine do all of the weight loading. Depending on whether the daemon already has weights or not, it will either exercise the regular model loading path (if it doesn't), or use the meta model initialization + IPC import (if it does).

We actually had an old prototype of the same architecture as this implementation. However, the weight daemon is actively holding a (CUDA) context because it does the loading of the weights - but there can be some classes of errors where holding onto a context causes the process to be poisoned, which means it needs to be killed (e.g. NVLink failure). If the daemon simply gives out CUDA allocations, no CUDA context is actually necessary allowing the daemon to resilient to more classes of failures. Another benefit is you don't need to deal with "fake sharding" in the weight daemon so it's less complex.

We've implemented an early version of active standby failover already within Dynamo, and one of the gaps here I see is how you would go about pause/resume with this architecture. Ultimately the secondary (we call this the "shadow" in Dynamo) would need to consume as few GPU resources as possible, so we would need to make sure the KV cache is unmapped, and free as many buffers as possible, while sharing the same set of weights. I'm wondering whether you have any thoughts on your approach to this problem with this architecture. One way to handle this is via a custom TMS backend: fzyzcjy/torch_memory_saver#79

The meta-model instantiation path is quite brittle. I see here that there are a few things that need to be hard-coded (some tensors are views of others and need to be rewired to the same storage, etc) that are in _rebuild_stale_views, among other issues (like some required tensors being tensor attributes but not registered as parameters/buffers etc). We are also running into similar problems, and a lot of the issues on this front are caused by replaying the host-side initialization. Is the plan to patch all of these instances one by one? Or should we have an integration test that allows "importing" from the daemon to always work?

Absolutely, let's collaborate on iterating this feature.

[liusy58]

/rerun-failed-ci

[ShangmingCai]

CI has passed: