Skip to content

ci-overview.md

Latest commit

 

History

History
192 lines (137 loc) · 9.41 KB

ci-overview.md

File metadata and controls

192 lines (137 loc) · 9.41 KB
Raw

Continuous Integration Overview

This page explains how TensorRT‑LLM's CI is organized and how individual tests map to Jenkins stages. Most stages execute integration tests defined in YAML files, while unit tests run as part of a merge‑request pipeline. The sections below describe how to locate a test and trigger the stage that runs it.

Table of Contents

  1. CI pipelines
  2. Test definitions
  3. Unit tests
  4. Jenkins stage names
  5. Finding the stage for a test
  6. Waiving tests
  7. Multi-GPU Tests
  8. Triggering CI Best Practices

CI pipelines

Pull requests do not start testing by themselves. Developers trigger the CI by commenting /bot run (optionally with arguments) on the pull request (see Pull Request Template for more details). That kicks off the merge-request pipeline (defined in jenkins/L0_MergeRequest.groovy), which runs unit tests and integration tests whose YAML entries specify stage: pre_merge. Once a pull request is merged, the post-merge pipeline runs the same jenkins/L0_MergeRequest.groovy as a separate Jenkins job. It runs the pre-merge stages plus the multi-GPU stages and every stage with Post-Merge in its name. In both pipelines, the test stages run in child jobs defined by jenkins/L0_Test.groovy, which maps each stage name to a test-db YAML file.

stage tags live in the YAML files under tests/integration/test_lists/test-db/. Searching those files for stage: pre_merge shows exactly which tests the merge-request pipeline covers.

Test definitions

Integration tests are listed under tests/integration/test_lists/test-db/. Most YAML files are named after the GPU or configuration they run on (for example l0_a100.yml). Some files, like l0_sanity_check.yml, use wildcards and can run on multiple hardware types. Entries contain conditions and a list of tests. Two important terms in each entry are:

  • stage: either pre_merge or post_merge.
  • backend: for example pytorch, cpp, or fmha. Grep the YAML files for backend: to see the values currently in use.

Example from l0_a30.yml:

      terms:
        stage: pre_merge
        backend: pytorch
  tests:
  - unittest/_torch/sampler/test_beam_search.py

Unit tests

Unit tests live under tests/unittest/. They are listed in the same test-db YAML files as integration tests, as entries starting with unittest/ (see the l0_a30.yml example above), and run in the jenkins/L0_Test.groovy stages mapped to those files, like any other test.

Jenkins stage names

jenkins/L0_Test.groovy maps stage names to these YAML files. For A100 the mapping includes:

    "A100X-PyTorch-1": ["a100x", "l0_a100", 1, 1],
    "A100X-PyTorch-Post-Merge-1": ["a100x", "l0_a100", 1, 1],

The array elements are: GPU type, YAML file (without extension), shard index, and total number of shards. Only tests with stage: post_merge from that YAML file are selected when a Post-Merge stage runs.

Stage names are unbracketed, with Post-Merge embedded directly in the name. Copy the name exactly as it appears in the Groovy map. Jenkins stage_list matches names without a * wildcard exactly, so a stale spelling such as A100X-Triton-[Post-Merge]-1 silently selects nothing. The mapping script's --stages option is stricter: it reports unknown names and near-match suggestions on stderr.

Finding the stage for a test

  1. Locate the test in the appropriate YAML file under tests/integration/test_lists/test-db/ and note its stage and backend values.
  2. Search jenkins/L0_Test.groovy for a stage whose YAML file matches (for example l0_a100) and whose name contains Post-Merge if the YAML entry uses stage: post_merge.
  3. The resulting stage name(s) are what you pass to Jenkins via the stage_list parameter when triggering a job.

Using test_to_stage_mapping.py

Manually searching YAML and Groovy files can be tedious. The helper script scripts/test_to_stage_mapping.py automates the lookup:

python scripts/test_to_stage_mapping.py --tests "unittest/_torch/sampler/test_beam_search.py"
python scripts/test_to_stage_mapping.py --tests test_beam_search
python scripts/test_to_stage_mapping.py --stages A100X-PyTorch-Post-Merge-1
python scripts/test_to_stage_mapping.py --test-list my_tests.txt
python scripts/test_to_stage_mapping.py --test-list my_tests.yml

The first two commands print the Jenkins stages that run the specified tests or patterns. Patterns are matched by substring, so partial test names are supported out of the box. The third lists every test executed in the given stage. When providing tests on the command line, quote each test string so the shell does not interpret the [ and ] characters as globs. Alternatively, store the tests in a newline‑separated text file or a YAML list and supply it with --test-list.

To run the same tests on your pull request, comment:

/bot run --stage-list "A100X-PyTorch-Post-Merge-1"

This executes the same tests that run post-merge for this hardware/backend.

Waiving tests

Sometimes a test is known to fail due to a bug or unsupported feature. Instead of removing it from the YAML test lists, add the test name to tests/integration/test_lists/waives.txt. Every CI run passes this file to pytest via --waives-file, so the listed tests are skipped automatically.

Each line contains the fully qualified test name followed by an optional SKIP (reason) marker. A full:GPU_TYPE/ prefix restricts the waive to a specific hardware family. Example:

accuracy/test_disaggregated_serving.py::TestDeepSeekV32Exp::test_auto_dtype[False] SKIP (https://nvbugs/6120535)
full:A100/accuracy/test_llm_api_pytorch_multimodal.py::TestExaone4_5_33B::test_auto_dtype[full_budget] SKIP (https://nvbugs/6422318)

Changes to waives.txt should include a bug link or brief explanation so other developers understand why the test is disabled.

Multi-GPU Tests

Running multi-GPU tests (--add-multi-gpu-test, --only-multi-gpu-test) requires the ci: full pre-merge approved label on the PR. Without this label the [Test-x86_64-Multi-GPU] Remote Run and [Test-SBSA-Multi-GPU] Remote Run stages will fail with an explanatory error.

To obtain the label, ask a member of NVIDIA/trt-llm-ci-approvers to apply it. Only members of that team can add the label; unauthorized additions are automatically removed by the Guard Full Pre-Merge Approval Label GitHub Actions workflow.

Once the label is present, re-trigger CI with the same command you used originally. For example:

# Run the normal pipeline plus multi-GPU stages
/bot run --add-multi-gpu-test

# Run only multi-GPU stages
/bot run --only-multi-gpu-test

Post-merge pipelines and GitLab MR builds are exempt from this label check.

Triggering CI Best Practices

Triggering Post-merge tests

Full /bot run --post-merge runs require the ci: post-merge approved PR label because they can consume substantial shared GPU resources. The label is intended to be applied by an active member of the NVIDIA/trt-llm-ci-approvers GitHub team. A GitHub workflow validates the label actor and normally removes invalid approvals. This is a best-effort resource governance guard, not a strict authorization boundary. The label remains in place when new commits are pushed and can be removed manually when the approval no longer applies.

When you only need to verify a handful of post-merge tests, specify exactly which stages to run:

/bot run --stage-list "stage-A,stage-B"

This runs only the stages listed. You can also add stages on top of the default pre-merge set:

/bot run --extra-stage "stage-A,stage-B"

Both options accept stage names and wildcard patterns defined in jenkins/L0_Test.groovy. Explicit Post-merge stage test doesn't need ci: post-merge approved PR label to run. However, the "*", "*Post-Merge*", and "*PerfSanity*" selectors require the same approval label, including when they appear in a comma-separated list. Equivalent escaped or repeated-star forms are treated the same. Other stage selectors, including explicit stage names and other limited wildcard patterns, retain their existing behavior.

Being selective keeps CI turnaround fast and conserves hardware resources.

Avoiding unnecessary --disable-fail-fast usage

Avoid habitually using --disable-fail-fast as it wastes scarce hardware resources. The CI system automatically reuses successful test stages when commits remain unchanged, and subsequent /bot run commands only retry failed stages. Overusing --disable-fail-fast keeps failed pipelines consuming resources (like DGX-H100s), increasing queue backlogs and reducing team efficiency.