Skip to content

Commit bb0f9e4

Browse files
michaelshinmichaelshin
authored
docs: add TensorRT-LLM General Inference walkthrough (#56)
1 parent 49458bd commit bb0f9e4

5 files changed

Lines changed: 164 additions & 1 deletion

File tree

‎docs/apps/inference.mdx‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,7 @@ CCluster provides pre-configured recipes for popular inference engines in the **
5656
- **SGLang** — `python3 -m sglang.launch_server --model-path <HF_MODEL_REPO> --host 0.0.0.0`
5757
- **LMDeploy** — `lmdeploy serve api_server <HF_REPO_NAME> --server-name 0.0.0.0`
5858
- **Ollama** — leave **Add command** empty. See the Ollama-specific note below.
59+
- **TensorRT-LLM** — `trtllm-serve <HF_REPO_NAME> --host 0.0.0.0 --port 8000`. See the TensorRT-LLM note below.
5960

6061
<Note>
6162
**Ollama** does not require a custom entrypoint. After the deployment is ready, you need to pull a model before it can serve requests:
@@ -69,6 +70,10 @@ curl -X POST https://<endpoint_url>/api/chat -d '{"model": "qwen2:1.5b", "messag
6970
```
7071
</Note>
7172

73+
<Tip>
74+
For a full TensorRT-LLM walkthrough, including config file mounts and endpoint testing, see [Deploying TensorRT-LLM](/resources/trtllm).
75+
</Tip>
76+
7277
### Deploying NVIDIA NIM containers
7378

7479
[NVIDIA NIM](https://developer.nvidia.com/nim) microservices are self-contained containers that bundle a model-optimized inference engine and download model weights automatically at startup, exposing an OpenAI-compatible API. NIM is not a pre-configured recipe — enter the image manually.

‎docs/resources/trtllm.mdx‎

Lines changed: 154 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,154 @@
1+
---
2+
title: 'Deploying TensorRT-LLM'
3+
description: 'Deploy TensorRT-LLM images on CCluster through General Inference'
4+
icon: 'gauge'
5+
---
6+
7+
[NVIDIA TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM) images run through [General Inference](/apps/inference) and expose an OpenAI-compatible API with `trtllm-serve`. This example walks through deploying `TinyLlama/TinyLlama-1.1B-Chat-v1.0` with a validated release image.
8+
9+
## Before you begin
10+
11+
Prepare the following:
12+
13+
- Access to a cluster with a suitable NVIDIA GPU hardware instance
14+
- A Hugging Face token if the model is gated or you need higher Hub rate limits
15+
- A Bearer token or client certificate for endpoint authentication
16+
17+
The image and model versions must be compatible. Review the [TensorRT-LLM support matrix](https://nvidia.github.io/TensorRT-LLM/reference/support-matrix.html) before using a different model or image tag.
18+
19+
## Configure the deployment
20+
21+
From **Create deployment**, select **Inference**, then configure the container:
22+
23+
| Field | Example value |
24+
|---|---|
25+
| **Add or Select Image** | `nvcr.io/nvidia/tensorrt-llm/release` |
26+
| **Container Tag** | `1.3.0rc22` |
27+
| **Container Port** | `8000` |
28+
| **Protocol** | `HTTP` |
29+
| **Health check path** | `/health` |
30+
31+
The image might appear as `nvidia/tensorrt-llm/release` in the prebuilt image list. The tag above is a validated example; use a tag that supports your selected model.
32+
33+
### Add command
34+
35+
The TensorRT-LLM release image does not start an inference server by itself. Under **Optional Details → Add command**, enter:
36+
37+
```bash
38+
/bin/bash -lc 'trtllm-serve TinyLlama/TinyLlama-1.1B-Chat-v1.0 --host 0.0.0.0 --port 8000 --config /etc/engine/config.yaml'
39+
```
40+
41+
Replace the model identifier with the Hugging Face model or TensorRT-LLM engine path you want to serve. Wrap the full `trtllm-serve` command in `/bin/bash -lc '...'` so the image shell initializes the TensorRT-LLM environment before the server starts.
42+
43+
| Argument | Why it matters |
44+
|---|---|
45+
| `/bin/bash -lc '...'` | Loads the image environment before running `trtllm-serve` |
46+
| `trtllm-serve <model>` | Starts the OpenAI-compatible server |
47+
| `--host 0.0.0.0` | Required so CCluster can reach the server |
48+
| `--port 8000` | Must match **Container Port** |
49+
| `--config /etc/engine/config.yaml` | Points at the mounted config file |
50+
51+
### Config file
52+
53+
Add a config file with these settings:
54+
55+
| Field | Value |
56+
|---|---|
57+
| **Filename** | `config.yaml` |
58+
| **Mount path** | `/etc/engine` |
59+
60+
Use the directory as the mount path, not the full filename. CCluster mounts the file at `/etc/engine/config.yaml`.
61+
62+
For the TinyLlama example, use:
63+
64+
```yaml
65+
model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
66+
max_batch_size: 8
67+
max_num_tokens: 2048
68+
max_seq_len: 2048
69+
trust_remote_code: true
70+
71+
kv_cache_config:
72+
free_gpu_memory_fraction: 0.8
73+
enable_block_reuse: true
74+
75+
enable_iter_perf_stats: true
76+
```
77+
78+
Tune `max_batch_size`, `max_seq_len`, and `free_gpu_memory_fraction` for your model and GPU memory. For TensorRT-LLM 1.3.x, keep these options at the top level; do not wrap them in `pytorch_backend_config`.
79+
80+
### Model credentials
81+
82+
Under **Optional Details → Environment Variables**, set `HF_TOKEN` to a Hugging Face token when the model is gated or private. You can store the token in a [Vault object](/resources/vault) instead of entering it directly.
83+
84+
Every endpoint must also be protected with a Bearer token, an mTLS client certificate, or both. See [Securing Endpoints](/resources/private).
85+
86+
From here the deployment follows the standard General Inference flow — select the cluster and hardware, click **Deploy**, and wait until the deployment is ready. Model download and GPU initialization can take several minutes. See [General Inference](/apps/inference) for details.
87+
88+
## Test the endpoint
89+
90+
Copy the endpoint URL and Bearer token from the deployment detail page:
91+
92+
```bash
93+
export ENDPOINT='https://<endpoint_url>'
94+
export CCLUSTER_TOKEN='<your_bearer_token>'
95+
96+
# Check readiness
97+
curl -sS -o /dev/null -w "%{http_code}\n" \
98+
-H "Authorization: Bearer $CCLUSTER_TOKEN" \
99+
"$ENDPOINT/health"
100+
101+
# Find the served model identifier
102+
curl -sS \
103+
-H "Authorization: Bearer $CCLUSTER_TOKEN" \
104+
"$ENDPOINT/v1/models"
105+
106+
# Send a chat completion request
107+
curl -sS "$ENDPOINT/v1/chat/completions" \
108+
-H "Authorization: Bearer $CCLUSTER_TOKEN" \
109+
-H "Content-Type: application/json" \
110+
-d '{
111+
"model": "TinyLlama/TinyLlama-1.1B-Chat-v1.0",
112+
"messages": [{"role": "user", "content": "Say hello."}],
113+
"max_tokens": 64
114+
}'
115+
```
116+
117+
Use the exact model `id` returned by `/v1/models` in subsequent requests. The first request can take longer while the runtime finishes warming up.
118+
119+
## Troubleshooting
120+
121+
| Symptom | Resolution |
122+
|---|---|
123+
| Deployment does not become ready although the server started | Confirm the command includes `--host 0.0.0.0`. |
124+
| Logs report `Is a directory: '.../config.yaml'` | Set **Mount path** to `/etc/engine` and **Filename** to `config.yaml`. |
125+
| Logs report unsupported `pytorch_backend_config` arguments | Remove that block and place supported options at the top level of the YAML file. |
126+
| API returns `401` | Send the deployment's Bearer token in the `Authorization` header. |
127+
| API returns `503` or `no healthy upstream` | Wait for the deployment to become ready, then check the host and config mount settings. |
128+
| Model download is slow or fails | Set `HF_TOKEN` and confirm that it can access the model. |
129+
130+
## What's next
131+
132+
<CardGroup cols={3}>
133+
<Card
134+
title="General Inference"
135+
icon="circle-nodes"
136+
href="/apps/inference"
137+
>
138+
Deploy and scale any containerized model on CCluster.
139+
</Card>
140+
<Card
141+
title="Securing Endpoints"
142+
icon="lock"
143+
href="/resources/private"
144+
>
145+
Protect your endpoints with Bearer tokens and mTLS certificates.
146+
</Card>
147+
<Card
148+
title="Managing Vault Objects"
149+
icon="key-skeleton-left-right"
150+
href="/resources/vault"
151+
>
152+
Store the registry credentials and API keys your deployments use.
153+
</Card>
154+
</CardGroup>

‎sidebars.js‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,7 @@ const sidebars = {
4040
{ type: 'doc', id: 'resources/hardware_instances', customProps: { icon: 'server' } },
4141
{ type: 'doc', id: 'resources/grpc', customProps: { icon: 'waypoints' } },
4242
{ type: 'doc', id: 'resources/nim', customProps: { icon: 'box-open-full' } },
43+
{ type: 'doc', id: 'resources/trtllm', customProps: { icon: 'gauge' } },
4344
{ type: 'doc', id: 'resources/private', customProps: { icon: 'lock' } },
4445
{ type: 'doc', id: 'resources/service_accounts', customProps: { icon: 'bot' } },
4546
{ type: 'doc', id: 'resources/json_and_tool', customProps: { icon: 'user-secret' } },

‎src/css/docs.css‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -113,7 +113,8 @@ aside[class*='docSidebarContainer'] > div {
113113

114114
nav.menu {
115115
padding: 1rem 1.25rem 1rem 0.75rem;
116-
overflow: hidden;
116+
overflow-x: hidden;
117+
overflow-y: auto;
117118
}
118119

119120
/* ── Navbar ──────────────────────────────────────────────────────────────── */

‎src/icons.js‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ import {
2222
ListChecks,
2323
Bot,
2424
Server,
25+
Gauge,
2526
} from 'lucide-react';
2627

2728
// Single source of truth for icon names → Lucide components, shared by the
@@ -54,6 +55,7 @@ const icons = {
5455
'list-checks': ListChecks,
5556
bot: Bot,
5657
server: Server,
58+
gauge: Gauge,
5759
};
5860

5961
export function renderIcon(name, {size = 16, strokeWidth = 1.75} = {}) {

0 commit comments

Comments
 (0)