Skip to content

Commit 085e4d5

Browse files
V2arKV2arK
authored
docs: move NVIDIA NIM guide to dedicated Resources page (#47)
Signed-off-by: Honglin Cao <hocao@nvidia.com>
1 parent c75903a commit 085e4d5

3 files changed

Lines changed: 99 additions & 48 deletions

File tree

‎docs/apps/inference.mdx‎

Lines changed: 4 additions & 48 deletions
Original file line numberDiff line numberDiff line change
@@ -71,55 +71,11 @@ curl -X POST https://<endpoint_url>/api/chat -d '{"model": "qwen2:1.5b", "messag
7171

7272
### Deploying NVIDIA NIM containers
7373

74-
[NVIDIA NIM](https://developer.nvidia.com/nim) microservices are self-contained containers that bundle a model-optimized inference engine and download model weights automatically at startup, exposing an OpenAI-compatible API. NIM is not a pre-configured recipe — enter the image manually with the settings below.
74+
[NVIDIA NIM](https://developer.nvidia.com/nim) microservices are self-contained containers that bundle a model-optimized inference engine and download model weights automatically at startup, exposing an OpenAI-compatible API. NIM is not a pre-configured recipe — enter the image manually.
7575

76-
NIM images are hosted on `nvcr.io`. Browse the catalog at [build.nvidia.com](https://build.nvidia.com/explore/discover) and copy the image path from the model's **Deploy** tab, for example:
77-
78-
```
79-
nvcr.io/nim/deepseek-ai/deepseek-r1-distill-qwen-32b:latest
80-
nvcr.io/nim/meta/llama-3.1-8b-instruct:latest
81-
```
82-
83-
Every NIM uses the same configuration:
84-
85-
| Field | Value |
86-
|---|---|
87-
| **Add or Select Image** | The `nvcr.io/nim/...` image path (entered manually) |
88-
| **Container Port** | `8000` |
89-
| **Protocol** | `HTTP` |
90-
| **Health check path** | `/v1/health/ready` |
91-
| **Image Registry Username** | `$oauthtoken` (literal string — not a variable) |
92-
| **Image Registry Password** | Your [NGC API Key](https://ngc.nvidia.com) |
93-
94-
Leave **Add command** empty — the image's default entrypoint starts the server. Under **Optional Details → Environment Variables**, also add:
95-
96-
| Key | Value |
97-
|---|---|
98-
| `NGC_API_KEY` | Your NGC API Key |
99-
100-
The NGC API Key is needed twice: the registry credentials let CCluster pull the image from `nvcr.io`, and the `NGC_API_KEY` environment variable lets the container download model weights on first boot. Without the environment variable, the container starts but fails to load the model.
101-
102-
<Note>
103-
NIM downloads model weights at startup, so the first boot can take several minutes for large models. The deployment stays in a pending state until the `/v1/health/ready` probe passes.
104-
</Note>
105-
106-
Once the deployment is ready, query it like any OpenAI-compatible endpoint. List the served model identifier first, then send requests:
107-
108-
```bash
109-
# Find the model identifier served by this NIM
110-
curl https://<endpoint_url>/v1/models \
111-
-H "Authorization: Bearer <your_bearer_token>"
112-
113-
# Chat completion
114-
curl https://<endpoint_url>/v1/chat/completions \
115-
-H "Authorization: Bearer <your_bearer_token>" \
116-
-H "Content-Type: application/json" \
117-
-d '{
118-
"model": "meta/llama-3.1-8b-instruct",
119-
"messages": [{"role": "user", "content": "Hello"}],
120-
"max_tokens": 256
121-
}'
122-
```
76+
<Tip>
77+
For the full NIM configuration and an end-to-end walkthrough, see [Deploying NVIDIA NIM](/resources/nim).
78+
</Tip>
12379

12480
## 2. Select the cluster and hardware to deploy
12581
By default, NVIDIA CCluster provides several managed clusters and GPU instances for you to deploy your inference containers.

‎docs/resources/nim.mdx‎

Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
---
2+
title: 'Deploying NVIDIA NIM'
3+
description: 'Deploy NVIDIA NIM microservices on CCluster through General Inference'
4+
icon: 'box-open-full'
5+
---
6+
7+
[NVIDIA NIM](https://developer.nvidia.com/nim) microservices are self-contained containers that bundle a model-optimized inference engine and download model weights automatically at startup, exposing an OpenAI-compatible API. NIM is not a pre-configured recipe on the [General Inference](/apps/inference) page — enter the image manually with the settings below.
8+
9+
## Find a NIM image
10+
11+
NIM images are hosted on `nvcr.io`. Browse the catalog at [build.nvidia.com](https://build.nvidia.com/explore/discover) and copy the image path from the model's **Deploy** tab. The catalog shows the image and tag joined with a colon (e.g. `nvcr.io/nim/meta/llama-3.1-8b-instruct:latest`) — on CCluster, split them into the **Add or Select Image** and **Container Tag** fields:
12+
13+
| Image | Tag |
14+
|---|---|
15+
| `nvcr.io/nim/deepseek-ai/deepseek-r1-distill-qwen-32b` | `latest` |
16+
| `nvcr.io/nim/meta/llama-3.1-8b-instruct` | `latest` |
17+
18+
You also need an [NGC API Key](https://ngc.nvidia.com) — it is used both to pull the image and to download model weights.
19+
20+
## Configure the deployment
21+
22+
From the **General Inference** deployment page, every NIM uses the same configuration:
23+
24+
| Field | Value |
25+
|---|---|
26+
| **Add or Select Image** | The `nvcr.io/nim/...` image path (entered manually, without the tag) |
27+
| **Container Tag** | The image tag (e.g. `latest`) |
28+
| **Container Port** | `8000` |
29+
| **Protocol** | `HTTP` |
30+
| **Health check path** | `/v1/health/ready` |
31+
| **Image Registry Username** | `$oauthtoken` (literal string — not a variable) |
32+
| **Image Registry Password** | Your [NGC API Key](https://ngc.nvidia.com) |
33+
34+
Leave **Add command** empty — the image's default entrypoint starts the server. Under **Optional Details → Environment Variables**, also add:
35+
36+
| Key | Value |
37+
|---|---|
38+
| `NGC_API_KEY` | Your NGC API Key |
39+
40+
The NGC API Key is needed twice: the registry credentials let CCluster pull the image from `nvcr.io`, and the `NGC_API_KEY` environment variable lets the container download model weights on first boot. Without the environment variable, the container starts but fails to load the model.
41+
42+
<Note>
43+
NIM downloads model weights at startup, so the first boot can take several minutes for large models. The deployment stays in a pending state until the `/v1/health/ready` probe passes.
44+
</Note>
45+
46+
From here the deployment follows the standard General Inference flow — select the cluster and hardware, click **Deploy**, and monitor the deployment until it is ready. See [General Inference](/apps/inference) for details. Check the model's page on [build.nvidia.com](https://build.nvidia.com/explore/discover) for the GPU and memory each NIM supports.
47+
48+
Once ready, query the endpoint like any OpenAI-compatible endpoint. List the served model identifier first, then send requests:
49+
50+
```bash
51+
# Find the model identifier served by this NIM
52+
curl https://<endpoint_url>/v1/models \
53+
-H "Authorization: Bearer <your_bearer_token>"
54+
55+
# Chat completion
56+
curl https://<endpoint_url>/v1/chat/completions \
57+
-H "Authorization: Bearer <your_bearer_token>" \
58+
-H "Content-Type: application/json" \
59+
-d '{
60+
"model": "meta/llama-3.1-8b-instruct",
61+
"messages": [{"role": "user", "content": "Hello"}],
62+
"max_tokens": 256
63+
}'
64+
```
65+
66+
<Tip>
67+
Every endpoint is protected — there are no public endpoints. Secure access with a Bearer token, an mTLS client certificate, or both. For mTLS, select **Make it a private endpoint?** and CCluster generates a TLS certificate that clients must present. See [Securing Endpoints](/resources/private) for details.
68+
</Tip>
69+
70+
## What's next
71+
72+
<CardGroup cols={3}>
73+
<Card
74+
title="General Inference"
75+
icon="circle-nodes"
76+
href="/apps/inference"
77+
>
78+
Deploy and scale any containerized model on CCluster.
79+
</Card>
80+
<Card
81+
title="Securing Endpoints"
82+
icon="lock"
83+
href="/resources/private"
84+
>
85+
Protect your endpoints with Bearer tokens and mTLS certificates.
86+
</Card>
87+
<Card
88+
title="Managing Vault Objects"
89+
icon="key-skeleton-left-right"
90+
href="/resources/vault"
91+
>
92+
Store the registry credentials and API keys your deployments use.
93+
</Card>
94+
</CardGroup>

‎sidebars.js‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,7 @@ const sidebars = {
3737
items: [
3838
{ type: 'doc', id: 'resources/custom_image', customProps: { icon: 'screwdriver-wrench' } },
3939
{ type: 'doc', id: 'resources/grpc', customProps: { icon: 'waypoints' } },
40+
{ type: 'doc', id: 'resources/nim', customProps: { icon: 'box-open-full' } },
4041
{ type: 'doc', id: 'resources/private', customProps: { icon: 'lock' } },
4142
{ type: 'doc', id: 'resources/service_accounts', customProps: { icon: 'bot' } },
4243
{ type: 'doc', id: 'resources/json_and_tool', customProps: { icon: 'user-secret' } },

0 commit comments

Comments
 (0)