Skip to content

trtllm.mdx

Latest commit

 

History

History
154 lines (116 loc) · 5.83 KB

trtllm.mdx

File metadata and controls

154 lines (116 loc) · 5.83 KB
Raw
title Deploying TensorRT-LLM
description Deploy TensorRT-LLM images on CCluster through General Inference
icon gauge

NVIDIA TensorRT-LLM images run through General Inference and expose an OpenAI-compatible API with trtllm-serve. This example walks through deploying TinyLlama/TinyLlama-1.1B-Chat-v1.0 with a validated release image.

Before you begin

Prepare the following:

  • Access to a cluster with a suitable NVIDIA GPU hardware instance
  • A Hugging Face token if the model is gated or you need higher Hub rate limits
  • A Bearer token or client certificate for endpoint authentication

The image and model versions must be compatible. Review the TensorRT-LLM support matrix before using a different model or image tag.

Configure the deployment

From Create deployment, select Inference, then configure the container:

Field Example value
Add or Select Image nvcr.io/nvidia/tensorrt-llm/release
Container Tag 1.3.0rc22
Container Port 8000
Protocol HTTP
Health check path /health

The image might appear as nvidia/tensorrt-llm/release in the prebuilt image list. The tag above is a validated example; use a tag that supports your selected model.

Add command

The TensorRT-LLM release image does not start an inference server by itself. Under Optional Details → Add command, enter:

/bin/bash -lc 'trtllm-serve TinyLlama/TinyLlama-1.1B-Chat-v1.0 --host 0.0.0.0 --port 8000 --config /etc/engine/config.yaml'

Replace the model identifier with the Hugging Face model or TensorRT-LLM engine path you want to serve. Wrap the full trtllm-serve command in /bin/bash -lc '...' so the image shell initializes the TensorRT-LLM environment before the server starts.

Argument Why it matters
/bin/bash -lc '...' Loads the image environment before running trtllm-serve
trtllm-serve <model> Starts the OpenAI-compatible server
--host 0.0.0.0 Required so CCluster can reach the server
--port 8000 Must match Container Port
--config /etc/engine/config.yaml Points at the mounted config file

Config file

Add a config file with these settings:

Field Value
Filename config.yaml
Mount path /etc/engine

Use the directory as the mount path, not the full filename. CCluster mounts the file at /etc/engine/config.yaml.

For the TinyLlama example, use:

model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
max_batch_size: 8
max_num_tokens: 2048
max_seq_len: 2048
trust_remote_code: true

kv_cache_config:
  free_gpu_memory_fraction: 0.8
  enable_block_reuse: true

enable_iter_perf_stats: true

Tune max_batch_size, max_seq_len, and free_gpu_memory_fraction for your model and GPU memory. For TensorRT-LLM 1.3.x, keep these options at the top level; do not wrap them in pytorch_backend_config.

Model credentials

Under Optional Details → Environment Variables, set HF_TOKEN to a Hugging Face token when the model is gated or private. You can store the token in a Vault object instead of entering it directly.

Every endpoint must also be protected with a Bearer token, an mTLS client certificate, or both. See Securing Endpoints.

From here the deployment follows the standard General Inference flow — select the cluster and hardware, click Deploy, and wait until the deployment is ready. Model download and GPU initialization can take several minutes. See General Inference for details.

Test the endpoint

Copy the endpoint URL and Bearer token from the deployment detail page:

export ENDPOINT='https://<endpoint_url>'
export CCLUSTER_TOKEN='<your_bearer_token>'

# Check readiness
curl -sS -o /dev/null -w "%{http_code}\n" \
  -H "Authorization: Bearer $CCLUSTER_TOKEN" \
  "$ENDPOINT/health"

# Find the served model identifier
curl -sS \
  -H "Authorization: Bearer $CCLUSTER_TOKEN" \
  "$ENDPOINT/v1/models"

# Send a chat completion request
curl -sS "$ENDPOINT/v1/chat/completions" \
  -H "Authorization: Bearer $CCLUSTER_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "TinyLlama/TinyLlama-1.1B-Chat-v1.0",
    "messages": [{"role": "user", "content": "Say hello."}],
    "max_tokens": 64
  }'

Use the exact model id returned by /v1/models in subsequent requests. The first request can take longer while the runtime finishes warming up.

Troubleshooting

Symptom Resolution
Deployment does not become ready although the server started Confirm the command includes --host 0.0.0.0.
Logs report Is a directory: '.../config.yaml' Set Mount path to /etc/engine and Filename to config.yaml.
Logs report unsupported pytorch_backend_config arguments Remove that block and place supported options at the top level of the YAML file.
API returns 401 Send the deployment's Bearer token in the Authorization header.
API returns 503 or no healthy upstream Wait for the deployment to become ready, then check the host and config mount settings.
Model download is slow or fails Set HF_TOKEN and confirm that it can access the model.

What's next

Deploy and scale any containerized model on CCluster. Protect your endpoints with Bearer tokens and mTLS certificates. Store the registry credentials and API keys your deployments use.