Skip to content

Commit 49458bd

Browse files
authored
docs: add NVIDIA Dynamo deployment guide (#55)
1 parent bf314d0 commit 49458bd

5 files changed

Lines changed: 167 additions & 0 deletions

File tree

‎docs/apps/dynamo.mdx‎

Lines changed: 166 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,166 @@
1+
---
2+
title: 'NVIDIA Dynamo'
3+
description: 'Deploy and operate NVIDIA Dynamo inference endpoints'
4+
icon: 'waypoints'
5+
---
6+
7+
NVIDIA Dynamo deployments are backed by a Dynamo Frontend and one or more GPU-powered vLLM workers. HTTP deployments expose an OpenAI-compatible API, while gRPC deployments expose the KServe v2 inference protocol. NVIDIA CCluster manages the Dynamo runtime, deployment lifecycle, TLS ingress, revisions, scaling configuration, status, and logs.
8+
9+
<Note>
10+
NVIDIA Dynamo support is currently in beta. It is available only on clusters where the Dynamo operator is enabled. If **Dynamo** is not available when you create a deployment, [request support](/resources/requesting_support).
11+
</Note>
12+
13+
The current integration uses an **aggregated** serving topology:
14+
15+
- One CPU-only **Frontend** accepts requests and routes them to the workers.
16+
- One or more **Worker** replicas run vLLM on the selected GPU hardware.
17+
- Every worker replica consumes one complete instance of the hardware type you select.
18+
- NVIDIA CCluster supplies the Dynamo runtime image. You select the model and hardware instead of providing a container image.
19+
20+
## Before you begin
21+
22+
Prepare the following:
23+
24+
- A Hugging Face model identifier, such as `Qwen/Qwen3-0.6B`.
25+
- Access to the model on Hugging Face. For gated or private models, accept the model license and provide a Hugging Face token that can download it.
26+
- A Bearer token or client certificate for endpoint authentication. Public, unauthenticated endpoints are not supported.
27+
- Enough GPU quota and cluster capacity for your maximum worker replica count.
28+
29+
## 1. Configure the endpoint
30+
31+
From **New Deployment**, choose **Dynamo**. The **Endpoint** step contains the required serving settings:
32+
33+
- **Endpoint Name** — a unique name containing lowercase letters, numbers, and hyphens. It must begin with a lowercase letter, end with a letter or number, and be no more than 20 characters.
34+
- **Model** — the Hugging Face model identifier loaded by the vLLM workers.
35+
- **Served Model Name** — an optional alias clients can send in the OpenAI `model` field. If you leave it empty, use the Hugging Face model identifier.
36+
- **Hugging Face Token** — required for gated and private models.
37+
- **Protocol** — `HTTP` for the OpenAI-compatible API or `gRPC` for the KServe v2 inference protocol.
38+
- **Private Endpoint** — select or create a Bearer token or client certificate.
39+
- **Annotations** — optional key/value metadata applied to the deployment components.
40+
41+
<Tip>
42+
For Bearer token and mTLS setup, see [Securing Endpoints](/resources/private). Do not place endpoint credentials in environment variables.
43+
</Tip>
44+
45+
<Frame>
46+
<img src="/images/dynamo_endpoint_configuration.png" alt="NVIDIA Dynamo Endpoint step with model, protocol, private endpoint, and annotation fields" style={{ borderRadius: '0.5rem' }} />
47+
</Frame>
48+
49+
## 2. Configure runtime options
50+
51+
The **Advanced** step includes optional runtime and observability settings:
52+
53+
- **Enable logging** — enabled by default. Container output is collected and displayed in the deployment's **Logs** tab.
54+
- **Enable Model Caching** — stores model files in a node-local persistent cache. Restarts and replicas scheduled on the same node can reuse the files instead of downloading them again.
55+
- **Extra Arguments** — additional vLLM worker arguments. During the beta, use this field only when directed by NVIDIA CCluster support; an unsupported value can prevent the worker from starting.
56+
- **Environment Variables** — additional variables passed to the Frontend and workers. Use shell-compatible names containing letters, numbers, and underscores, beginning with a letter or underscore.
57+
58+
<Frame>
59+
<img src="/images/dynamo_advanced_configuration.png" alt="NVIDIA Dynamo Advanced step with logging, model caching, extra arguments, and environment variables" style={{ borderRadius: '0.5rem' }} />
60+
</Frame>
61+
62+
## 3. Select hardware and scaling
63+
64+
In the **Hardware** step, choose a cluster and GPU hardware type. The selection applies to each worker replica: for example, if the selected hardware is `8 x H100`, every worker uses one `8 x H100` instance.
65+
66+
Configure the worker count with these fields:
67+
68+
| Setting | Description |
69+
| --- | --- |
70+
| **Min Replicas** | Minimum number of GPU worker replicas. Must be at least `1`; Dynamo deployments cannot scale to zero. |
71+
| **Max Replicas** | Maximum number of GPU worker replicas. Set it equal to **Min Replicas** for a fixed-size deployment. |
72+
| **Concurrency** | Target number of in-flight requests per worker. Required when **Max Replicas** is greater than **Min Replicas**. |
73+
| **Cooldown Period** | Scale-down stabilization time in seconds, from `0` to `3600`. |
74+
75+
<Note>
76+
Use a fixed-size deployment with **Min Replicas** equal to **Max Replicas**. Dynamic autoscaling is not currently supported for customer workloads.
77+
</Note>
78+
79+
<Frame>
80+
<img src="/images/dynamo_hardware_scaling.png" alt="NVIDIA Dynamo Hardware step with GPU hardware choices and fixed worker replica settings" style={{ borderRadius: '0.5rem' }} />
81+
</Frame>
82+
83+
NVIDIA CCluster evaluates GPU quota against **Max Replicas**. For a supported fixed-size deployment, set **Min Replicas** and **Max Replicas** to the same value, and make sure the cluster has enough capacity for every worker to consume the complete selected hardware instance.
84+
85+
Click **Deploy**. Model download and worker startup can take several minutes, especially for large models that are not already cached.
86+
87+
## 4. Monitor the deployment
88+
89+
The deployments list and revision table report Dynamo pods by component instead of combining them into one replica count:
90+
91+
- **Frontend** — the request-routing component. A standard deployment has one Frontend pod.
92+
- **Worker** — the GPU-backed vLLM replicas controlled by the scaling settings.
93+
94+
For example, `Frontend (1 / 1 replicas)` and `Worker (2 / 2 replicas)` mean that the routing component and both GPU workers are ready. The overall deployment can remain not ready when the Frontend is unhealthy, even if all workers are running, because the endpoint cannot accept traffic without it.
95+
96+
Open the deployment to access:
97+
98+
- **Detail** — endpoint, model, hardware, scaling configuration, component health, and revision history.
99+
- **Logs** — logs for a selected Frontend or Worker pod and revision. Logs are available only for revisions with logging enabled.
100+
- **Terminal** — an interactive shell for an active deployment.
101+
- **Usage** — deployment resource usage.
102+
103+
Editing a Dynamo deployment creates a new revision and starts an operator-managed rolling update. You can inspect historical revision status and logs, clone a deployment, or roll back to an earlier revision. Pausing or resuming the whole deployment is supported; pausing only an in-progress Dynamo rollout is not.
104+
105+
## 5. Call the endpoint
106+
107+
For a Bearer-protected HTTP deployment, copy the endpoint hostname and Bearer token from the **Detail** tab. Send requests to the OpenAI-compatible chat completions endpoint:
108+
109+
```bash
110+
curl "https://<endpoint_url>/v1/chat/completions" \
111+
-H "Authorization: Bearer <your_bearer_token>" \
112+
-H "Content-Type: application/json" \
113+
-d '{
114+
"model": "<model-or-served-model-name>",
115+
"messages": [
116+
{
117+
"role": "user",
118+
"content": "Explain why low-latency inference matters."
119+
}
120+
],
121+
"max_tokens": 128,
122+
"stream": false
123+
}'
124+
```
125+
126+
For a certificate-protected endpoint, configure your HTTP client with the downloaded client certificate instead of the `Authorization` header. See [Securing Endpoints](/resources/private).
127+
128+
If you selected `gRPC`, connect to `<endpoint_url>:443` with TLS and use Dynamo's KServe v2 `ModelInfer` or `ModelStreamInfer` API. This is different from the direct vLLM gRPC API used by General Inference deployments. See the [NVIDIA Dynamo KServe gRPC frontend guide](https://docs.nvidia.com/dynamo/latest/components/frontend/frontend-guide) for the protocol and [Securing Endpoints](/resources/private) for authentication.
129+
130+
## Current scope
131+
132+
The beta integration currently supports:
133+
134+
- Aggregated serving with a Dynamo Frontend and vLLM workers.
135+
- Fixed-size worker deployments. Dynamic autoscaling is not currently supported for customer workloads.
136+
- HTTP OpenAI-compatible and KServe gRPC endpoints.
137+
- Bearer token or mTLS endpoint protection.
138+
- Revision-aware status, logs, editing, cloning, and rollback.
139+
140+
The current integration has a single Frontend and does not expose Frontend high availability, disaggregated prefill/decode serving, custom Dynamo runtime images, config-file mounts, or scale-to-zero. For Dynamo architecture and features outside the NVIDIA CCluster integration, see the [NVIDIA Dynamo documentation](https://docs.nvidia.com/dynamo/latest/).
141+
142+
## What's next
143+
144+
<CardGroup cols={3}>
145+
<Card
146+
title="Securing Endpoints"
147+
icon="lock"
148+
href="/resources/private"
149+
>
150+
Protect your endpoint with a Bearer token or mTLS client certificate.
151+
</Card>
152+
<Card
153+
title="NVIDIA Dynamo Frontend"
154+
icon="waypoints"
155+
href="https://docs.nvidia.com/dynamo/components/frontend"
156+
>
157+
Review Dynamo's OpenAI-compatible HTTP and KServe gRPC interfaces.
158+
</Card>
159+
<Card
160+
title="Get Support"
161+
icon="headset"
162+
href="/resources/requesting_support"
163+
>
164+
Ask about Dynamo availability, capacity, or deployment failures.
165+
</Card>
166+
</CardGroup>

‎sidebars.js‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ const sidebars = {
1616
collapsed: false,
1717
items: [
1818
{ type: 'doc', id: 'apps/llm', customProps: { icon: 'messages' } },
19+
{ type: 'doc', id: 'apps/dynamo', customProps: { icon: 'waypoints' } },
1920
{ type: 'doc', id: 'apps/inference', customProps: { icon: 'circle-nodes' } },
2021
{ type: 'doc', id: 'apps/compute', customProps: { icon: 'microchip' } },
2122
{ type: 'doc', id: 'apps/jobs', customProps: { icon: 'list-checks' } },
211 KB
Loading
179 KB
Loading
221 KB
Loading

0 commit comments

Comments
 (0)