Skip to content

AGENTS.md

Latest commit

 

History

History
233 lines (183 loc) · 14 KB

AGENTS.md

File metadata and controls

233 lines (183 loc) · 14 KB
Raw

AGENTS.md

TensorRT-LLM: open-source library for optimized LLM inference on NVIDIA GPUs. Python and C++ codebase with a PyTorch execution path.

If a CLAUDE.local.md file exists alongside this file, read and respect it — it contains developer-specific overrides that supplement this shared guidance.

Rules (Read First)

CRITICAL (YOU MUST):

  • Read and follow CODING_GUIDELINES.md for ALL code changes (C++ and Python)
  • NVIDIA copyright header on ALL new files (update year on modified files)
  • git commit -s (DCO sign-off required). Never attribute AI tools in sign-off line. Always rely on git to do the sign off instead of directly adding sign off in commit message.
  • Do not add co-authors to the git commit message unless explicitly instructed to do so by the user.
  • pre-commit hooks run on commit — if files are modified by hooks, re-stage and commit again
  • LLM args or nested-config changes must run python3 scripts/generate_llm_args_golden_manifest.py and commit tensorrt_llm/usage/llm_args_golden_manifest.json; new fields require telemetry/privacy CODEOWNER approval
  • Changes to serving YAML fields, aliases, defaults, types, or constraints (including nested, disaggregated, and VisualGen configs) must regenerate and inspect the editor JSON Schemas with python3 scripts/generate_trtllm_serve_schemas.py, using a TensorRT-LLM build matching the checkout. Update the generator's YAML-specific adaptations when configuration annotations alone do not capture the change, and add regression coverage for the affected accepted/rejected values. Run pytest -m cpu_only tests/unittest/llmapi/test_trtllm_serve_config_schemas.py. Commit the generated tensorrt_llm/schemas/*.json snapshots; never hand-edit them. Version bumps in tensorrt_llm/version.py also require regeneration: the snapshots include the package version in their $id and title. python3 scripts/generate_trtllm_serve_schemas.py --check and the CPU schema tests reject missing or stale snapshots. Wheels and HTML docs publish these same files.
  • When adding or renaming a public model architecture identifier, update tensorrt_llm/usage/architecture_allowlist.py with its exact identifier. It must be publicly documented by the upstream model provider or in docs/source/models/supported-models.md; never add private, customer-specific, or arbitrary user-supplied names
  • PR title format: [JIRA/NVBUG/None][type] description (e.g., [TRTLLM-5516][perf] optimize cuda graph padding)
  • Put a change's rationale and history (why it was made, what it replaces, ticket/PR references) in the PR description, not in code comments. Code comments should explain the code as it stands for a future reader, not narrate how it got there.
  • Set LLM_MODELS_ROOT env var when running tests that need model weights

Common Commands

Task Command
Unit tests pytest tests/unittest/
C++ component tests cmake --build cpp/build_RelWithDebInfo --target common-tests (also runtime-tests, executor-tests, batch-manager-tests, kernel-tests)
C++ test scheduling ctest --test-dir cpp/build_RelWithDebInfo -L common --output-on-failure (build the selected tests first)
Specific test pytest tests/unittest/llmapi/test_llm_args.py
Pattern match pytest tests/unittest -k "test_llm_args"
Integration tests LLM_MODELS_ROOT=/path/to/models pytest tests/integration/defs/...
Serve model trtllm-serve <hf_model> --port 8000
Serve with config trtllm-serve <hf_model> --config config.yaml
Benchmark trtllm-bench --model <hf_model> throughput --dataset <path>
Find CI stage for test python scripts/test_to_stage_mapping.py --tests "test_name"

Installation & Build

Building TensorRT-LLM requires Docker and may involve compiling C++ components. See the Installation Guide for pre-built release containers and pip install, build from source for development builds, and Container Images for information about the container images.

Reference Configs

examples/configs/database/ contains pareto-optimized serving configurations across multiple models, GPUs, ISL/OSL combinations, and concurrency levels. Use these as starting points for deployment and benchmarking rather than hand-tuning parameters. See deployment guides for model-specific walkthroughs.

Architecture

See architecture diagram for the full Mermaid diagram.

Backends

Backend Status Entry Point Key Path
PyTorch Default TorchLlmArgs _torch/pyexecutor/ → PyExecutor → PyTorch Engine

Shared C++ Core (via Nanobind)

Both backends share these C++ components:

  • Scheduling pipeline: Scheduler → BatchManager (in-flight batching) → KV Cache Manager
  • Decoding pipeline: Decoder (token generation orchestration) → Sampling

Request Flow

HuggingFace Model → LLM API → PyTorch Executor
    → Scheduler → Model Forward → Decoder → Sampling → Generated Tokens

Serving

  • trtllm-serve: OpenAI-compatible REST + gRPC server, supports all backends
  • Disaggregated serving: separates prefill (context) and decode (generation) across GPUs
    • KV cache exchange via NIXL (default), UCX, or MPI

Key Files

File Role
tensorrt_llm/llmapi/llm.py Main API entry point
tensorrt_llm/llmapi/llm_args.py Complete configuration schema (Pydantic)
tensorrt_llm/llmapi/llm_utils.py Model loading, model-specific default overrides
tensorrt_llm/models/modeling_utils.py Base classes for all models (PretrainedConfig, PretrainedModel)
tensorrt_llm/executor/executor.py Execution abstraction (GenerationExecutor)
tensorrt_llm/models/automodel.py Auto-discovery and model registry
tensorrt_llm/_torch/models/ PyTorch backend model implementations (distinct from the top-level models/ package)
tensorrt_llm/_torch/attention/ATTENTION_DEVELOPER_GUIDE.md Attention, MLA, backend families, sparse backends, metadata contracts, and KV-cache behavior - read before modifying anything under tensorrt_llm/_torch/attention/
tensorrt_llm/_torch/moe/fused_moe/MOE_DEVELOPER_GUIDE.md MoE architecture, backends, communication, development patterns — read before modifying MoE code
CODING_GUIDELINES.md C++ and Python coding standards (referenced throughout, must read before contributing)

Design Patterns

Pattern Key Points
Config hierarchy BaseLlmArgs → TorchLlmArgs, model-specific defaults override generics, Pydantic validation
Model architecture Each model: Config (inherits PretrainedConfig) + ForCausalLM (inherits PretrainedModel)
Model defaults Architecture-specific overrides in llm_utils.py (attention kernels, quant, spec decoding, cache)
Attention backends TorchLlmArgs.attn_backend selects kernel: TRTLLM (default), FlashInfer, FlashAttention
Distributed execution Tensor/pipeline parallelism via Mapping class, multiple backends (MPI, Ray, RPC)
Auto-discovery Models self-register via automodel.py, resolved by HF config architectures field

VisualGen

VisualGen is a vertical alongside LLM for Diffusion-Transformer (DiT)-based image/video generation (text-to-image, text-to-video, image-to-video). It is not an LLM backend — it has its own engine, args, params, and outputs — but shares ops and kernels with the PyTorch backend where it makes sense (attention, quantization, parallelism).

Key entry points:

  • Public Python API: from tensorrt_llm import VisualGen, VisualGenArgs, VisualGenParams.
  • Serving CLI: trtllm-serve <HF id> --visual_gen_args <YAML path>.

Key files:

  • tensorrt_llm/_torch/visual_gen/ENGINEERING_CRITERIA.md: Engineering criteria for any change under tensorrt_llm/visual_gen/ or tensorrt_llm/_torch/visual_gen/ — API discipline, feature/test/lossy-vs-lossless requirements, examples & docs rules. Read before modifying anything in those trees.
  • tensorrt_llm/visual_gen/: VisualGen public Python API. User-facing surface — before modifying anything here, pause and confirm with the user that a public API change is actually intended; do not infer it from the surrounding task.
  • tensorrt_llm/_torch/visual_gen/: VisualGen internal implementation. All non-user-facing code belongs here.

Anti-Patterns / Gotchas

  • Pre-commit modifies files in-place — if hooks fail, files are already modified. Re-stage (git add) and commit again.
  • Protected APIs exist — changes to LLM API signatures will fail tests/unittest/api_stability tests. Get code owner review.
  • Integration tests need GPUs + models — always set LLM_MODELS_ROOT and ensure GPU access. Unit tests don't.
  • Copyright year — update to current year when modifying existing files; add full header to new files.
  • Avoid broad exception handling — catch specific exceptions, not bare except: (see CODING_GUIDELINES.md).
  • One concern per PR — avoid scope creep. If a PR touches unrelated areas, split it.
  • User-facing configuration classes - when editing or defining any user-facing configuration classes (particularly BaseLlmArgs or any class used in its fields), you MUST follow the Pydantic guidelines in CODING_GUIDELINES.md.

Development Workflow

  1. Set up build environment (see installation docs)
  2. Make changes following CODING_GUIDELINES.md
  3. Test locally with pytest

Branching policy and PRs

  • The main repository (upstream) is located at https://github.com/NVIDIA/TensorRT-LLM/
  • Branches should always be pushed to the user-specified fork (usually origin)
  • If pushing fails to due pre-push pre-commits hooks getting updated, just re-push immediately
  • PRs should be opened on the main repository
    • Target main unless fixing a release branch bug
    • See CONTRIBUTING.md for full PR policies

GitHub CLI authentication (GH_CONFIG_DIR)

The gh CLI uses ~/.config/gh by default for authentication. Different GitHub hosts or forks may require a different config directory. Before running any gh command (e.g., gh pr create, gh api, gh pr comment):

  1. Check if the user has specified a custom GH_CONFIG_DIR (e.g., in CLAUDE.local.md or environment). If so, use it.
  2. If not explicitly set, default to ~/.config/gh; do not ask for confirmation.
  3. Prefix all gh commands with the resolved config dir: GH_CONFIG_DIR=<path> gh ...
  4. If the command fails due to missing authentication or the wrong GitHub host/account, report the failure and ask for the correct GH_CONFIG_DIR.

CI / Testing

See CI overview for full details.

Layer Location Notes
Unit tests tests/unittest/ Run in pre-merge CI; some tests require GPU
API stability tests/unittest/api_stability/ Protects committed API signatures
Integration tests tests/integration/defs/ Requires GPU + LLM_MODELS_ROOT
Test lists tests/integration/test_lists/test-db/ Per-GPU YAML files (l0_a10.yml, l0_h100.yml, etc.)
Test waives tests/integration/test_lists/waives.txt Skip known-failing tests with NVBug links
Performance See benchmarking guide trtllm-bench and trtllm-serve benchmarks

New C++ unit tests should use add_tllm_gtest with explicit SOURCES, LINK_LIBRARIES, OWNER, and scheduling LABELS. Request FULL_STACK explicitly for integration tests and REQUIRES_GPU for tests that need a GPU. google-tests continues to build all C++ tests; component aggregate targets build migrated tests.

Advisory semantic review

The bot-to-bot semantic review workflow is disabled. See .github/semantic-review.md for its current status and historical implementation details.

Vendor source tools and promotion

Use the tools under scripts/vendor/ to manage locked vendor sources, prepare lock-only promotions, and optionally monitor source-update PRs. Read Vendored Sources for commands, contributor metadata, and operator guidance before changing vendor state or publishing promotions.

Triggering CI

CI is triggered by posting comments on the PR. Basic commands:

  • /bot run — trigger the standard CI pipeline
  • /bot run --disable-fail-fast — run all stages even if earlier ones fail (only add when explicitly needed)

For a full list of up-to-date bot commands, post /bot help as a PR comment and check the bot's reply.

Trouble Shooting

  • Use TLLM_LOG_LEVEL_BY_MODULE to enable per-module log filtering (e.g., "debug:_torch,runtime;info:serve"); see Module-Level Logging for details.

Key Documentation

Topic Path
Coding guidelines CODING_GUIDELINES.md
Architecture overview docs/source/developer-guide/overview.md
PyTorch backend docs/source/torch/arch_overview.md
Adding a new model docs/source/torch/adding_new_model.md
Disaggregated serving docs/source/features/disagg-serving.md
Speculative decoding docs/source/features/speculative-decoding.md
Quantization docs/source/features/quantization.md
Parallelism strategies docs/source/features/parallel-strategy.md
KV cache docs/source/features/kvcache.md
API change guidelines docs/source/developer-guide/api-change.md
Feature compatibility matrix docs/source/features/feature-combination-matrix.md
Supported models docs/source/models/supported-models.md
Deployment guides docs/source/deployment-guide/
Examples & customization docs/source/examples/
Performance analysis docs/source/developer-guide/perf-analysis.md