[https://nvbugs/6939002][fix] Tell the agentx aiperf fork from PyPI aiperf before skipping its install by chenfeiz0326 · Pull Request #20482 · NVIDIA/TensorRT-LLM

chenfeiz0326 · 2026-10-16T15:43:48Z

Description

The GB300 AgentX perf-sanity lane disagg_upload-e2e-gb300_deepseek-v4-pro-dspark_agentx_con1456_ctx3_dep8_gen1_dep16_eplb0_dspark5_ccb-NIXL has failed on every release/1.3 post-merge run since build 29, and on main since build 3020, before sending a single request:

[agentx] aiperf + deps already usable; skipping install
aiperf profile --scenario inferencex-agentx-mvp ...
Unknown option: --scenario. Did you mean --slice-duration?
RuntimeError: aiperf exited 1 and wrote no .../profile_export_aiperf.json

Root cause

agentx_client.ensure_aiperf skips installing the pinned agentx-harness build when aiperf_is_usable() succeeds. That check only imported aiperf (plus its deps) and checked that datasets is 4.x or newer. requirements-dev.txt pins the upstream PyPI aiperf==0.8.0, which has the same import name. While requirements.txt pinned datasets==3.1.0, the check always failed and the fork was installed. #19909 (release/1.3) and #20175 (main) unpinned datasets (now 4.7.0). Since then the check passes on the PyPI aiperf, the fork is never installed, and PyPI aiperf rejects --scenario.

CI history matches. On release/1.3, post-merge 28 (without #19909) passed and 29 onward (with it) failed. On main, 3018 passed, while 3020 and 3021 (with #20175) failed.

Fix

aiperf_is_usable() now also requires the fork's scenario registry to resolve inferencex-agentx-mvp, and logs the reason when a check fails. With the PyPI aiperf present, the pinned fork is installed as before.
The scenario name is a module constant shared with build_aiperf_cmd.
New CPU unit test tests/unittest/scripts/test_perf_agentx_aiperf_probe.py, already collected by unittest/scripts in l0_cpu.yml. On the old client, 5 of its 6 cases fail.
Removes this bug's waiver.

Test Coverage

Original case, oci-jhb, 10x GB300, CI image pytorch-26.09-py3-sbsa-ubuntu24.04-skip-tritondevel-202610130623-20146, release/1.3 post-merge 34 wheel:

	commit	result
before	`41c0ec8` (origin/release/1.3)	install skipped -> `Unknown option: --scenario`; pytest 1 failed
after	this PR	fork installed; `health OK: 79373 requests over 3629.9s, submission_valid=true`; pytest 1 passed

tests/unittest/scripts: the new test passes, and test_subprocess_timeouts passes.

Fixes https://nvbugs/6939002

🤖 Generated with Claude Code

Dev Engineer Review

aiperf_is_usable() now rejects an aiperf install unless it imports the required modules, registers inferencex-agentx-mvp, and uses datasets 4 or newer. It logs the failure reason.
build_aiperf_cmd uses the shared scenario constant. Verify that the probe and command stay aligned when the scenario changes.
Review findings and severity counts were not supplied. Reported test results were not independently verified.

QA Engineer Review

Added six tests covering usable fork stubs, PyPI-style installs, a missing scenario, old datasets, a missing module, and command scenario selection.
The supplied change summary reports removal of the waiver for the GB300 DeepSeek V4 Pro disaggregated-upload test. The returned repository output showed no diff for waives.txt, so that specific change could not be confirmed from the inspected diff.
The tests cover the probe’s main success and failure paths. Coverage verdict: sufficient for the probe behavior. Test-run results remain unverified.

Per-File QA Perspective

tests/integration/defs/perf/agentx_client.py: Verify that an upstream PyPI aiperf install fails the probe and triggers fork installation. Verify that command construction passes the scenario checked by the probe.
tests/unittest/scripts/test_perf_agentx_aiperf_probe.py: Covers probe success and failure paths and scenario selection. It is a unit-test file, not an integration test-list entry.
tests/integration/test_lists/waives.txt: The supplied change summary describes removal of a performance-test waiver. The inspected output did not establish that file’s diff; verify the intended test is no longer skipped.

coderabbitai · 2026-10-16T15:52:05Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository: NVIDIA/TensorRT-LLM/.coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 7cae9316-bfe8-46ed-8aa6-7682d420c5c9

📥 Commits

Reviewing files that changed from the base of the PR and between e1d7a72 and 9031b8b.

📒 Files selected for processing: 1

tests/unittest/scripts/test_perf_agentx_aiperf_probe.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

Walkthrough

The AgentX client checks required imports, scenario registration, and the datasets version before use. Its command uses a shared scenario identifier. CPU-only tests cover probe outcomes and command construction. The waiver list no longer skips one GB300 DeepSeek V4 Pro performance test.

Changes

AgentX performance test readiness

Layer / File(s)	Summary
aiperf probe and scenario selection `tests/integration/defs/perf/agentx_client.py`, `tests/unittest/scripts/test_perf_agentx_aiperf_probe.py`	The probe checks required imports, scenario registration, and the `datasets` major version. It logs the final stderr line when a check fails. The command uses `AGENTX_SCENARIO`. CPU-only tests cover probe outcomes and command construction.
GB300 performance test waiver `tests/integration/test_lists/waives.txt`	The skip entry for the GB300 DeepSeek V4 Pro disaggregated upload performance test is removed.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~15 minutes

Change: Bug fix

Merge Risk: ⚪ Minimal · up to 9031b

No concrete issue currently blocks merging; complete the normal CI checks.

Pre-merge checks: 4 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 28.57% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 14 functions across 2 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 4 passed

Check name	Status	Explanation
Title check		The title clearly identifies the AgentX aiperf fork detection fix, includes the valid NVBugs ID and fix type, and matches the primary changes.
Description check		The description clearly explains the failure, root cause, fix, affected lane, test coverage, and validation results. It does not include the template's PR Checklist section, but the description is oth…
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

Full details: Description check

Explanation

The description clearly explains the failure, root cause, fix, affected lane, test coverage, and validation results. It does not include the template's PR Checklist section, but the description is otherwise substantially complete.

Fix all pre-merge checks with AI

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create a new PR

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

…iperf before skipping its install agentx_client.ensure_aiperf skips installing the pinned agentx-harness build when aiperf_is_usable() succeeds. That check only imported `aiperf` and checked the datasets major version. requirements-dev.txt pins the upstream PyPI aiperf==0.8.0, which shares the import name, so once the CI image's datasets became 4.x the check passed on the PyPI build. The client then ran `aiperf profile --scenario inferencex-agentx-mvp`, which PyPI aiperf rejects ("Unknown option: --scenario"), and the lane failed before sending a request. The check now also requires the fork's scenario registry to resolve the inferencex-agentx-mvp scenario, and logs why a check failed. A PyPI aiperf now leads to the pinned fork being installed as before. Remove this bug's waiver. Signed-off-by: chenfeiz0326 <203214996+chenfeiz0326@users.noreply.github.com>

chenfeiz0326 · 2026-10-18T08:46:02Z

/bot run --disable-fail-fast

chenfeiz0326 · 2026-10-18T08:50:33Z

Requesting a code-owner review. This fixes NVBug 6939002: the GB300 AgentX DSpark con1456 perf-sanity lane has failed on every release/1.3 post-merge since build 29. Since datasets was unpinned (#19909), agentx_client.aiperf_is_usable() accepted the PyPI aiperf==0.8.0 from requirements-dev.txt as the agentx-harness fork, so the fork was never installed and aiperf rejected --scenario. The check now also requires the fork's inferencex-agentx-mvp scenario. The original 10-node GB300 case failed before the change and passed after it (79,373 requests, submission_valid=true); details are in the PR description. The branch was rebased onto release/1.3 to resolve a waives.txt conflict; the code is unchanged. blossom-ci was triggered just now.

tensorrt-cicd · 2026-10-18T08:52:05Z

PR_Github #78707 [ run ] triggered by Bot. Commit: e1d7a72 Link to invocation

tensorrt-cicd · 2026-10-18T10:01:04Z

PR_Github #78707 [ run ] completed with state FAILURE. Commit: e1d7a72
/LLM/release-1.3/L0_MergeRequest_PR pipeline #320 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

tests/unittest/conftest.py has an autouse fixture that requests capfd, and pytest refuses capsys together with capfd, so five of the probe tests errored at setup in the CI unit-test lane. Signed-off-by: chenfeiz0326 <203214996+chenfeiz0326@users.noreply.github.com>

chenfeiz0326 · 2026-10-18T11:26:09Z

The previous blossom-ci run (L0_MergeRequest_PR #320) failed in CPU-Generic-x86-1-cbts: five of the new test_perf_agentx_aiperf_probe.py tests errored at setup with cannot use capsys and capfd at the same time, because the autouse cuda_error_early_quit(capfd) fixture in tests/unittest/conftest.py already uses capfd. 9031b8b switches the test to capfd; with that conftest loaded in the CI image the file now passes 15/15. The Build-SBSA abort in that run (Queue task was cancelled after the artifact upload) was CI infra, not this change. Re-running CI.

chenfeiz0326 · 2026-10-18T11:26:11Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-10-18T11:33:54Z

PR_Github #78710 [ run ] triggered by Bot. Commit: 9031b8b Link to invocation

tensorrt-cicd · 2026-10-18T12:43:23Z

PR_Github #78710 [ run ] completed with state SUCCESS. Commit: 9031b8b
/LLM/release-1.3/L0_MergeRequest_PR pipeline #321 completed with status: 'UNSTABLE'

CI Report

⚠️ Multi-GPU Label Required:
Multi-GPU tests require the ci: full pre-merge approved label on this PR. Either:

Wait for the PR to be fully approved — the label is added automatically once approval is complete. Having unresolved open comments is fine, or
If needed, ask a member of NVIDIA/trt-llm-ci-approvers to add the label manually.
Then re-trigger CI with the same bot command (no rebase needed).

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

chenfeiz0326 · 2026-10-18T13:32:00Z

CI status for 9031b8b (L0_MergeRequest_PR #321, UNSTABLE): no test failed. All 12 test_perf_agentx_aiperf_probe.py cases pass on CPU-Generic-x86-1-cbts and CPU-Generic-arm-1-cbts, and the other pre-merge stages are green. The only red stage is [Test-SBSA-Multi-GPU] Blocked, which waits for the ci: full pre-merge approved label (it is added on code-owner approval). After that I will re-run /bot run --disable-fail-fast.

github-actions · 2026-10-19T00:42:44Z

Automatically added "ci: full pre-merge approved" because this PR has satisfied the required GitHub review approvals. Unresolved review conversations and other required checks remain independent merge requirements.

chenfeiz0326 · 2026-10-19T04:35:31Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-10-19T04:43:35Z

PR_Github #78798 [ run ] triggered by Bot. Commit: 9031b8b Link to invocation

tensorrt-cicd · 2026-10-19T05:50:08Z

PR_Github #78798 [ run ] completed with state SUCCESS. Commit: 9031b8b
/LLM/release-1.3/L0_MergeRequest_PR pipeline #330 completed with status: 'SUCCESS'

CI Report

Link to invocation

chenfeiz0326 requested review from a team as code owners October 16, 2026 15:43

chenfeiz0326 requested review from EmmaQiaoCh, brnguyen2, fredricz-20070104, mlefeb01 and schetlur-nv October 16, 2026 15:43

github-actions Bot assigned chenfeiz0326 Oct 16, 2026

chenfeiz0326 force-pushed the repair-bot-v2/nvbug-6939002-r7 branch from 420b731 to e1d7a72 Compare October 18, 2026 08:45

BowenFu approved these changes Oct 19, 2026

View reviewed changes

trtllm-agent added the ci: full pre-merge approved label Oct 19, 2026

chenfeiz0326 merged commit 869e311 into NVIDIA:release/1.3 Oct 19, 2026
15 checks passed

Conversation

chenfeiz0326 commented Oct 16, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Root cause

Fix

Test Coverage

Dev Engineer Review

QA Engineer Review

Per-File QA Perspective

Uh oh!

coderabbitai Bot commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

chenfeiz0326 commented Oct 18, 2026

Uh oh!

chenfeiz0326 commented Oct 18, 2026

Uh oh!

tensorrt-cicd commented Oct 18, 2026

Uh oh!

tensorrt-cicd commented Oct 18, 2026

Uh oh!

chenfeiz0326 commented Oct 18, 2026

Uh oh!

chenfeiz0326 commented Oct 18, 2026

Uh oh!

tensorrt-cicd commented Oct 18, 2026

Uh oh!

tensorrt-cicd commented Oct 18, 2026

Uh oh!

chenfeiz0326 commented Oct 18, 2026

Uh oh!

github-actions Bot commented Oct 19, 2026

Uh oh!

chenfeiz0326 commented Oct 19, 2026

Uh oh!

tensorrt-cicd commented Oct 19, 2026

Uh oh!

tensorrt-cicd commented Oct 19, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

chenfeiz0326 commented Oct 16, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Oct 16, 2026 •

edited

Loading