Skip to content

Add TensorRT-LLM Transfer, Action, and Distilled guides #310

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ishovkun wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Add TensorRT-LLM Transfer, Action, and Distilled guides #310

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
f5deb1c
Add TensorRT-LLM audio and V2V guide
ishovkun Aug 7, 2026
7a07a15
Add TensorRT-LLM Transfer, Action, and Distilled guides
ishovkun Aug 7, 2026
a454afd
Update TensorRT-LLM Action and Transfer recipes
ishovkun Aug 28, 2026
9ceef65
Fix TensorRT-LLM guardrail cookbook setup
ishovkun Aug 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1141,17 +1141,21 @@ We are building examples that show Cosmos 3 Super/Nano/Edge capabilities end to
| --- | --- | --- | --- | --- |
| Generator (audiovisual) with Diffusers | Generator | Text-to-image, plus text-to-video and image-to-video each with or without synchronized sound, via `Cosmos3OmniPipeline`. | [Notebook](cookbooks/cosmos3/generator/audiovisual/run_with_diffusers.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/audiovisual/run_with_diffusers.ipynb) |
| Generator (audiovisual) with Cosmos Framework | Generator | Text-to-image, plus text-to-video and image-to-video each with sound on or off, through the `cosmos_framework.scripts.inference` entrypoint. | [Notebook](cookbooks/cosmos3/generator/audiovisual/run_with_cosmos_framework.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/audiovisual/run_with_cosmos_framework.ipynb) |
| Generator (audiovisual) with TensorRT-LLM | Generator | Text-to-image, text-to-video, and image-to-video against an OpenAI-compatible TensorRT-LLM VisualGen server. | [Notebook](cookbooks/cosmos3/generator/audiovisual/run_with_trt_llm.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/audiovisual/run_with_trt_llm.ipynb) |
| Generator (audiovisual) with TensorRT-LLM | Generator | Text-to-image, text-to-video and image-to-video with or without synchronized audio, video-to-video, and the published four-step T2I/I2V students against an OpenAI-compatible TensorRT-LLM VisualGen server. | [Notebook](cookbooks/cosmos3/generator/audiovisual/run_with_trt_llm.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/audiovisual/run_with_trt_llm.ipynb) |
| Generator (audiovisual) with vLLM-Omni | Generator | Text-to-image, text-to-video, image-to-video, and video-to-video, with supported sound modes, against an OpenAI-compatible vLLM-Omni server. | [Notebook](cookbooks/cosmos3/generator/audiovisual/run_with_vllm_omni.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/audiovisual/run_with_vllm_omni.ipynb) |
| Generator (audiovisual) with NIM | Generator | Text2Video and Image2Video only, against the prebuilt `Cosmos3-Generator` NIM; requests use `POST /v1/infer` and decode JSON `b64_video` responses. | [Notebook](cookbooks/cosmos3/generator/audiovisual/run_with_nim.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/audiovisual/run_with_nim.ipynb) |
| Generator (audiovisual) with SGLang | Generator | Text-to-image, plus text-to-video and image-to-video each with sound on or off, against an OpenAI-compatible SGLang server. | [Notebook](cookbooks/cosmos3/generator/audiovisual/run_with_sglang.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/audiovisual/run_with_sglang.ipynb) |
| Forward dynamics with Cosmos Framework | Generator | Forward dynamics: action-conditioned future-observation prediction for AV, DROID, UMI, and human hand pose, through the `cosmos_framework.scripts.inference` entrypoint. | [Notebook](cookbooks/cosmos3/generator/action/run_fd_with_cosmos_framework.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/action/run_fd_with_cosmos_framework.ipynb) |
| Forward dynamics with TensorRT-LLM | Generator | Forward dynamics from a checked-in AV image and action trajectory, returning rollout and action tensors through the VisualGen API. | [Notebook](cookbooks/cosmos3/generator/action/run_fd_with_trt_llm.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/action/run_fd_with_trt_llm.ipynb) |
| Forward dynamics with vLLM-Omni | Generator | Forward dynamics: action-conditioned future-observation prediction for AV, DROID, UMI, and human hand pose, against an OpenAI-compatible vLLM-Omni server. | [Notebook](cookbooks/cosmos3/generator/action/run_fd_with_vllm_omni.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/action/run_fd_with_vllm_omni.ipynb) |
| Forward dynamics with SGLang | Generator | Forward dynamics: action-conditioned future-observation prediction for AV, DROID, and UMI, against an OpenAI-compatible SGLang server. | [Notebook](cookbooks/cosmos3/generator/action/run_fd_with_sglang.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/action/run_fd_with_sglang.ipynb) |
| Inverse dynamics with Cosmos Framework | Generator | Inverse dynamics: ego-motion trajectory prediction from input AV video, through the `cosmos_framework.scripts.inference` entrypoint. | [Notebook](cookbooks/cosmos3/generator/action/run_id_with_cosmos_framework.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/action/run_id_with_cosmos_framework.ipynb) |
| Inverse dynamics with TensorRT-LLM | Generator | Inverse dynamics from a checked-in AV observation clip, returning reconstructed video and the predicted trajectory in a tensor payload. | [Notebook](cookbooks/cosmos3/generator/action/run_id_with_trt_llm.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/action/run_id_with_trt_llm.ipynb) |
| Inverse dynamics with vLLM-Omni | Generator | Inverse dynamics: ego-motion trajectory prediction from input AV video, against an OpenAI-compatible vLLM-Omni server. | [Notebook](cookbooks/cosmos3/generator/action/run_id_with_vllm_omni.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/action/run_id_with_vllm_omni.ipynb) |
| Inverse dynamics with SGLang | Generator | Inverse dynamics: ego-motion trajectory prediction from input AV video, against an OpenAI-compatible SGLang server. | [Notebook](cookbooks/cosmos3/generator/action/run_id_with_sglang.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/action/run_id_with_sglang.ipynb) |
| Action policy with TensorRT-LLM | Generator | DROID policy inference from a concatenated three-camera first frame, returning a rollout and predicted action chunk through the VisualGen API. | [Notebook](cookbooks/cosmos3/generator/action/run_policy_with_trt_llm.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/action/run_policy_with_trt_llm.ipynb) |
| Transfer with Cosmos Framework | Generator | Video transfer: edge, blur, depth, segmentation, and world-scenario controls with captions, through the `cosmos_framework.scripts.inference` entrypoint. | [Notebook](cookbooks/cosmos3/generator/transfer/run_video_transfer_with_cosmos_framework.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/transfer/run_video_transfer_with_cosmos_framework.ipynb) |
| Transfer with TensorRT-LLM | Generator | Edge, blur, depth, segmentation, and WSM transfer using inline precomputed controls and synchronous encoded-video responses; raw uploads are used only for server-derived edge/blur. | [Notebook](cookbooks/cosmos3/generator/transfer/run_video_transfer_with_trt_llm.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/transfer/run_video_transfer_with_trt_llm.ipynb) |
| Transfer with vLLM-Omni | Generator | Video transfer: edge, blur, depth, segmentation, and world-scenario controls with captions, against an OpenAI-compatible vLLM-Omni server. | [Notebook](cookbooks/cosmos3/generator/transfer/run_video_transfer_with_vllm_omni.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/transfer/run_video_transfer_with_vllm_omni.ipynb) |
| Reasoner with Cosmos Framework | Reasoner | Text and image reasoning: detailed captioning, robot task planning, 2D grounding, describe-anything, and action-trajectory prompts, through the `cosmos_framework.scripts.inference` entrypoint. | [Notebook](cookbooks/cosmos3/reasoner/run_with_cosmos_framework.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/reasoner/run_with_cosmos_framework.ipynb) |
| Reasoner with vLLM | Reasoner | Image and video reasoning: captioning, temporal localization, embodied reasoning, common-sense reasoning, 2D grounding, describe-anything, action CoT, driving scenes, physical-plausibility, and situation understanding, against an OpenAI-compatible vLLM server (Cosmos3-Super on 4 GPUs by default; switch to Nano per the cookbook README). | [Notebook](cookbooks/cosmos3/reasoner/run_with_vllm.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/reasoner/run_with_vllm.ipynb) |
Expand Down
89 changes: 72 additions & 17 deletions cookbooks/cosmos3/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ backend you want to run and follow that one section.
| --- | --- | --- |
| [Cosmos Framework](#cosmos-framework) | Native PyTorch inference, launched with `torchrun` | Reasoner, Generator (Audiovisual, Action, **Transfer**) |
| [Diffusers](#diffusers) | Direct generation with `Cosmos3OmniPipeline` | Generator (Audiovisual) |
| [TensorRT-LLM Generator](#tensorrt-llm-generator) | OpenAI-compatible VisualGen server (image/video generation) | Generator (Audiovisual) |
| [TensorRT-LLM Generator](#tensorrt-llm-generator) | OpenAI-compatible VisualGen server (image/video/audio/action/transfer generation) | Generator (Audiovisual, Action, **Transfer**) |
| [TensorRT-LLM Reasoner](#tensorrt-llm-reasoner) | OpenAI-compatible image/video reasoning server | Reasoner |
| [Transformers](#transformers) | Hugging Face Transformers inference | Reasoner |
| [vLLM](#vllm) | OpenAI-compatible reasoning server (image/video understanding) | Reasoner |
Expand Down Expand Up @@ -169,9 +169,15 @@ uv pip install --torch-backend=cu130 \
## TensorRT-LLM Generator

OpenAI-compatible **VisualGen** server for Generator audiovisual text-to-image,
text-to-video, and image-to-video examples. Cosmos3 support was added in TensorRT-LLM PR
[#14824](https://github.com/NVIDIA/TensorRT-LLM/pull/14824); use a
TensorRT-LLM checkout or package that includes that change.
text-to-video, image-to-video, video-to-video, synchronized audio, Transfer, and
Action examples, including the published four-step T2I and I2V students.
Initial Cosmos3 support was added in TensorRT-LLM PR
[#14824](https://github.com/NVIDIA/TensorRT-LLM/pull/14824), synchronized audio
in [#14827](https://github.com/NVIDIA/TensorRT-LLM/pull/14827), and
video-to-video in [#16155](https://github.com/NVIDIA/TensorRT-LLM/pull/16155),
Transfer in [#16394](https://github.com/NVIDIA/TensorRT-LLM/pull/16394), and
Action in [#17325](https://github.com/NVIDIA/TensorRT-LLM/pull/17325).
Use a TensorRT-LLM checkout or package that includes those changes.

Install TensorRT-LLM following its upstream documentation.

Expand All @@ -182,7 +188,7 @@ Cosmos3 VisualGen change before it is available in your installed package or
release image.

```bash
apt-get update && apt-get -y install git git-lfs
apt-get update && apt-get -y install ffmpeg git git-lfs
git lfs install

git clone https://github.com/NVIDIA/TensorRT-LLM.git
Expand Down Expand Up @@ -214,8 +220,10 @@ explicitly disable guardrails before starting the server:

```bash
pip install cosmos_guardrail==0.3.0
# If needed by your OpenCV stack:
# pip uninstall opencv-python
# On headless servers without libGL.so.1, replace the OpenCV wheel pulled in by
# cosmos_guardrail with the matching headless build:
pip uninstall -y opencv-python
pip install opencv-python-headless==5.0.0.93
```

Set the TensorRT-LLM source root for the shared VisualGen config YAMLs:
Expand All @@ -242,16 +250,63 @@ torchrun --nproc_per_node=4 -m tensorrt_llm.commands.serve \
--port "$COSMOS3_TRTLLM_PORT"
```

The server exposes `/health`, `/v1/videos/generations`, `/v1/videos`, and
`/v1/images/generations`. The audiovisual notebook uses the validated video
generation endpoint for text-to-image, text-to-video, and image-to-video. Cosmos3
text-to-image is sent as a one-frame video request, matching the TensorRT-LLM
Cosmos3 pipeline; the notebook sends it as `num_frames=1`, `seconds=1`, and
`fps=8` to satisfy the video request schema while preserving a single generated
frame. Requests send Cosmos3 controls through `extra_params`,
so use a TensorRT-LLM build that includes the Cosmos3 VisualGen API schema.
The notebook sets request-level `max_sequence_length=2048` for longer structured
JSON prompts.
**Four-step distilled T2I** (single GPU; 1024×1024, one-frame warmup):

```bash
trtllm-serve nvidia/Cosmos3-Super-Text2Image-4Step \
--visual_gen_args "$TRTLLM_ROOT/examples/visual_gen/configs/cosmos3-t2i-1gpu.yaml" \
--port "$COSMOS3_TRTLLM_PORT"
```

**Four-step distilled I2V** (single GPU; default 1280×720, 189-frame shape):

```bash
trtllm-serve nvidia/Cosmos3-Super-Image2Video-4Step \
--port "$COSMOS3_TRTLLM_PORT"
```

Action uses the Nano launch above. For DROID policy inference, replace the model
with `nvidia/Cosmos3-Nano-Policy-DROID` and keep the one-GPU Nano config.

The server exposes `/health`, the blocking `/v1/videos/sync`, the asynchronous
`/v1/videos`, and `/v1/images/generations`. The older
`/v1/videos/generations` spelling is a deprecated alias of `/v1/videos/sync`.
The audiovisual notebook uses the validated video
generation endpoint for text-to-image, text-to-video, image-to-video,
video-to-video, and synchronized audio. Cosmos3 text-to-image is sent as a
one-frame video request, matching the TensorRT-LLM Cosmos3 pipeline; the notebook
sends it as `num_frames=1`, `seconds=1`, and `fps=8` to satisfy the video request
schema while preserving a single generated frame. Image-to-video and
video-to-video upload their reference media as multipart `input_reference`;
TensorRT-LLM classifies the reference by content. Synchronized audio is enabled
with `enable_audio: true` in `extra_params` and is muxed into the output video.
Keep `ffmpeg` on the server `PATH`: without it, TensorRT-LLM falls back to a
video-only AVI encoder and cannot preserve generated audio. Requests send
Cosmos3 controls through `extra_params`, so use a TensorRT-LLM build that includes
the Cosmos3 VisualGen API schema. The notebook sets request-level
`max_sequence_length=4096` for longer structured JSON prompts.

Transfer uses the synchronous `/v1/videos/sync` route. For server-derived edge
or blur, upload the raw source video as multipart `input_reference` and set the
corresponding `extra_params` hint to `true`. For a precomputed edge, blur,
depth, segmentation, or WSM control, base64-encode the control inside its hint;
no `input_reference` is needed. The server decodes inline media to bytes at the
HTTP boundary. TensorRT-LLM uses `use_guardrails` for its per-request safety
switch; `guardrails`, `control_path`, and other vLLM-Omni-only names are not
interchangeable.

Action requests use the same synchronous route and upload an image or video as
`input_reference`. Because an action trajectory cannot be represented in MP4 or
AVI, `format=auto` resolves to `safetensors`; the payload contains named `video`,
`action`, and `frame_rate` tensors. The asynchronous `/v1/videos` route also
supports this payload: poll `GET /v1/videos/{id}`, then download it from
`GET /v1/videos/{id}/content`.

The distilled checkpoints own their four-step stochastic schedules and have
classifier-free guidance baked into their weights. Leave `num_inference_steps`
and `guidance_scale` unset; conflicting values are rejected. Also leave
`use_system_prompt` unset for distilled I2V so its checkpoint-declared default
is applied.

## TensorRT-LLM Reasoner

Expand Down
Loading