[TRTLLM-16460][feat] agent-flow: trtllm_modeling_v2 workflow for gated ModelingV2 onboarding and perf tuning

AGENTS.md

@@ -193,6 +193,21 @@ For a full list of up-to-date bot commands, post `/bot help` as a PR comment and
 
 - Use `TLLM_LOG_LEVEL_BY_MODULE` to enable per-module log filtering (e.g., `"debug:_torch,runtime;info:serve"`); see [Module-Level Logging](docs/source/developer-guide/overview.md#module-level-logging) for details.
 
+**Environment failures that read as model failures.** Each of these presents as a broken
+model or a broken build, and none of them is. Check them before debugging modeling code:
+
+| symptom | cause | fix |
+|---|---|---|
+| `MPI_ERR_SPAWN: could not spawn` | the container runtime forbids `MPI_Comm_spawn`, which the default LLM API path uses to start workers (normal under pyxis/enroot) | start one task per rank under `trtllm-llmapi-launch` so the ranks exist before the engine; `TLLM_WORKER_USE_SINGLE_PROCESS=1` only at world size 1 |
+| `Impossible to fit in any sequence in kvCache` | the engine inherited the checkpoint's `max_position_embeddings` (262144 is common) and sized the cache for it | pass `--max_seq_len` / `--max_batch_size` / `--kv_cache_fraction` explicitly |
+| `NVRTC ... could not open source file "cuda.h"` | usually the venv's `bin` missing from `PATH`; separately, the generation FMHA kernel takes the NVRTC path unless the KV dtype is `E2m1` (grep `options.mDtypeKv != tg::Dtype::E2m1`) | check `PATH` first; `--kv_cache_dtype fp8` takes the non-NVRTC branch |
+| the engine hangs at startup and never returns | a wrong launch substrate hangs rather than raises — typically an MPI connect-back on an interface that does not carry it | probe with a deadline rather than waiting |
+| the import resolves to a different checkout | `/root/.local` is shared across container runs on some clusters | `PYTHONNOUSERSITE=1` at runtime — but **not** during the native build, which needs the user site for `cutlass_library` |
+
+A longer list, with the wrong diagnoses recorded alongside the right ones, is in
+[`agent-flow/agent_flow/workflows/trtllm_modeling_v2/TROUBLESHOOTING.md`](agent-flow/agent_flow/workflows/trtllm_modeling_v2/TROUBLESHOOTING.md).
+It is keyed on verbatim error text, so search it for the string you saw.
+
 ## Key Documentation
 
 | Topic | Path |

agent-flow/README.md

@@ -297,4 +297,5 @@ block, before `run`: it writes each role's composed system prompt verbatim to
 handed in, since that is what the agents are built from (`AgentTeamWorkflow`
 appends its MCP-tool protocol to the bundle it is given). The `perf-analyze`
 and `perf-optimize` CLIs use it to snapshot `<workspace>/prompts/` at every
-launch.
+launch; `trtllm-modeling-v2` does the same from preflight, once the launch
+environment its prompts carry is resolved.

agent-flow/agent_flow/backends/claude_code.py

@@ -6,11 +6,12 @@
 import subprocess
 import sys
 import tempfile
-from contextlib import asynccontextmanager
+from contextlib import asynccontextmanager, contextmanager
+from contextvars import ContextVar
 from importlib.metadata import PackageNotFoundError
 from importlib.metadata import version as _pkg_version
 from pathlib import Path
-from typing import Any, AsyncIterator
+from typing import Any, AsyncIterator, Callable, Iterator
 
 import claude_agent_sdk
 from claude_agent_sdk import (
@@ -370,13 +371,114 @@ def _claude_sdk_version() -> str:
         return ""
 
 
+#: A ``claude`` binary every layer launches instead of the SDK's own choice.
+CLI_PATH_ENV = "AGENT_FLOW_CLAUDE_CLI"
+
+
+def _version_of(cli_path: str) -> tuple[int, ...]:
+    """``claude --version`` of ``cli_path`` as a comparable tuple; empty when unreadable."""
+    try:
+        out = subprocess.run(
+            [cli_path, "--version"], capture_output=True, text=True, timeout=_CLI_VERSION_TIMEOUT_S
+        )
+    except (OSError, subprocess.SubprocessError):
+        return ()
+    token = out.stdout.strip().split()[0] if out.returncode == 0 and out.stdout.strip() else ""
+    try:
+        return tuple(int(part) for part in token.split("."))
+    except ValueError:
+        return ()
+
+
+def _cli_override() -> str | None:
+    """The ``claude`` binary layers launch, when it is not the SDK's bundled one.
+
+    ``$AGENT_FLOW_CLAUDE_CLI`` names one outright. Otherwise the ``claude``
+    on ``PATH`` is used when it is newer than the bundled binary: an
+    organization can require a minimum CLI version, and the bundled one
+    only moves when the SDK is upgraded, so it is refused while the
+    user's own CLI runs. ``None`` keeps the SDK's choice.
+    """
+    named = os.environ.get(CLI_PATH_ENV, "").strip()
+    if named:
+        return named
+    global _CLI_OVERRIDE_CACHE
+    if _CLI_OVERRIDE_CACHE is None:
+        bundled = Path(claude_agent_sdk.__file__).parent / "_bundled" / "claude"
+        on_path = shutil.which("claude")
+        choice = ""
+        if on_path and bundled.is_file():
+            path_version = _version_of(on_path)
+            if path_version and path_version > _version_of(str(bundled)):
+                choice = on_path
+        _CLI_OVERRIDE_CACHE = choice
+    return _CLI_OVERRIDE_CACHE or None
+
+
+_CLI_OVERRIDE_CACHE: str | None = None
+
+
+def launched_cli() -> tuple[str, str]:
+    """The ``claude`` binary a layer launches here, and its version; ``""`` for what is unknown."""
+    return _find_claude_cli() or "", _claude_cli_version()
+
+
+#: Given each line a CLI started in this context writes to stderr; see :func:`cli_stderr_to`.
+_CLI_STDERR_LISTENER: ContextVar[Callable[[str], None] | None] = ContextVar(
+    "claude_cli_stderr_listener", default=None
+)
+
+
+@contextmanager
+def cli_stderr_to(listener: Callable[[str], None]) -> Iterator[None]:
+    """Hand ``listener`` every stderr line of a CLI started inside the block, too.
+
+    The lines still reach this process's stderr, and a listener set by an
+    enclosing block still gets them. A layer called in the block's thread
+    starts its CLI in the block's context, so the lines of one turn reach the
+    listener of that turn and of no other running beside it. A listener that
+    raises does not stop the CLI's stderr from being read.
+    """
+    outer = _CLI_STDERR_LISTENER.get()
+
+    def both(line: str) -> None:
+        if outer is not None:
+            _tell(outer, line)
+        _tell(listener, line)
+
+    token = _CLI_STDERR_LISTENER.set(both)
+    try:
+        yield
+    finally:
+        _CLI_STDERR_LISTENER.reset(token)
+
+
+def _tell(listener: Callable[[str], None], line: str) -> None:
+    try:
+        listener(line)
+    except Exception:  # noqa: BLE001 - an observer cannot fail the CLI it reads
+        pass
+
+
+def _forward_stderr(line: str) -> None:
+    """Print a CLI stderr line, so a start-up failure is diagnosable, and pass it to the listener."""
+    print(f"[claude-cli stderr] {line}", file=sys.stderr, flush=True)
+    listener = _CLI_STDERR_LISTENER.get()
+    if listener is not None:
+        _tell(listener, line)
+
+
 def _find_claude_cli() -> str | None:
-    """Locate the ``claude`` CLI binary the SDK would actually launch.
+    """Locate the ``claude`` CLI binary the SDK will actually launch.
 
-    Mirrors the SDK's lookup order — bundled binary first, then ``PATH``.
-    Returns ``None`` when nothing is found so the caller can simply omit
-    the CLI version from the rendered string.
+    The override (:func:`_cli_override`) first, then the SDK's own lookup
+    order — bundled binary first, then ``PATH``. Returns ``None`` when
+    nothing is found so the caller can simply omit the CLI version from
+    the rendered string.
     """
+    override = _cli_override()
+    if override:
+        return override
     bundled = Path(claude_agent_sdk.__file__).parent / "_bundled" / "claude"
     if bundled.is_file():
         return str(bundled)
@@ -576,14 +678,15 @@ async def create_client(
             cwd=work_dir,
             sandbox={"enabled": False},
             permission_mode="bypassPermissions",
+            cli_path=_cli_override(),
             hooks=hooks,
             disallowed_tools=[
                 *(disallowed_tools or []),
                 *(f"Skill({name})" for name in self._disabled_skills),
             ],
             # Forward the CLI's stderr so startup failures are diagnosable
             # instead of vanishing with the subprocess.
-            stderr=lambda line: print(f"[claude-cli stderr] {line}", file=sys.stderr, flush=True),
+            stderr=_forward_stderr,
         )
 
         async with ClaudeSDKClient(options=options) as sdk_client:

agent-flow/agent_flow/workflows/trtllm_modeling_v2/.gitignore

@@ -0,0 +1,5 @@
+# Placeholders a sandboxed agent session leaves in its working directory.
+# Nothing in this package is configuration for a session that runs here: a run
+# materializes its skills into its own workspace.
+.mcp.json
+.claude/