Skip to content

Commit 8344885

Browse files
committed
Fix trtllm rc28 API-drift breakage, regenerate uv.lock
Retested on GB200 (SLURM job 3126032) after the fixes: all 44 trtllm unit tests pass (was 41 passed / 3 failed). - tests/.../test_trtllm_backend.py: force the manual finalize fallback in test_collective_refit_runs_at_async_engine_boundary via `monkeypatch.setattr(backend.WorkerExtension, "finalize_weight_update", None, raising=False)`. rc28's WorkerExtension now has finalize_weight_update (rc21 didn't), so trtllm_backend.py's _finalize_weight_update silently switched to that fast path and the test's hardcoded call-order assertion (written only for the fallback) started failing. Ported verbatim from the equivalent fix already landing in #4122 (which adds the real finalize_weight_update-aware FP8 refit logic later) — this commit only makes the test deterministic across TRT-LLM versions, it does not implement anything new. - nemo_rl/models/generation/trtllm/trtllm_backend.py: import control_action_decorator from tensorrt_llm.executor.ray.utils (the rc28 location) with a fallback to the old tensorrt_llm._ray_utils path. The old path still works but emits "will be removed in a future release"; fixing now avoids a silent break on the next bump. - uv.lock: regenerated against the bumped ref (was still rc21-era). Resolves cleanly (529 packages), confirms tensorrt/tensorrt-cu13/ tensorrt-cu13-bindings/tensorrt-cu13-libs are no longer pulled in (matches upstream dropping its own tensorrt pin). Note: it resolves torch and nvidia-cudnn-frontend via this repo's override-dependencies (torch==2.11.0+cu130, nvidia-cudnn-frontend==1.25.0) rather than what TRT-LLM's own dependencies actually declare (torch>=2.13.0, nvidia-cudnn-frontend>=1.27.0) — uv's override mechanism forces the version project-wide without validating the sub-dependency is actually satisfied. This is the same conflict already flagged in 3rdparty/TensorRT-LLM-workspace/pyproject.toml's "KNOWN UNRESOLVED CONFLICTS" comment; still unresolved, not addressed by this commit. Signed-off-by: Erin Ho <14718778+hchings@users.noreply.github.com>
1 parent bcf611d commit 8344885

3 files changed

Lines changed: 709 additions & 1493 deletions

File tree

‎nemo_rl/models/generation/trtllm/trtllm_backend.py‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,13 @@
3030

3131
import torch
3232
import zmq
33-
from tensorrt_llm._ray_utils import control_action_decorator
33+
34+
try:
35+
# tensorrt_llm._ray_utils is the pre-rc28 location; it now re-exports from
36+
# here with a FutureWarning ("will be removed in a future release").
37+
from tensorrt_llm.executor.ray.utils import control_action_decorator
38+
except ImportError:
39+
from tensorrt_llm._ray_utils import control_action_decorator
3440
from tensorrt_llm.llmapi.rlhf_utils import WorkerExtension
3541

3642
from nemo_rl.models.policy.utils import (

‎tests/unit/models/generation/trtllm/test_trtllm_backend.py‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -94,6 +94,13 @@ def test_collective_refit_runs_at_async_engine_boundary(
9494
from nemo_rl.models.generation.trtllm import trtllm_backend as backend
9595

9696
extension, module, model, model_loader, engine = _extension(backend)
97+
# Force the manual finalize fallback so the asserted call order is
98+
# deterministic. On an installed TRT-LLM that actually has
99+
# WorkerExtension.finalize_weight_update, _finalize_weight_update takes
100+
# that fast path instead and never drives process/post individually.
101+
monkeypatch.setattr(
102+
backend.WorkerExtension, "finalize_weight_update", None, raising=False
103+
)
97104
call_order = []
98105

99106
def packed_consumer(*, iterator, group, src, post_unpack_func):

0 commit comments

Comments
 (0)