|
| 1 | +--- |
| 2 | +name: trtllm-modeling-v2 |
| 3 | +description: Launch and operate this repo's trtllm-modeling-v2 workflow, which brings a HuggingFace checkpoint up as a gated ModelingV2 target (catalog entries and an assembled target, proven by boot, accuracy and acceptance gates at every requested parallel member, with optional features such as chunked prefill, block reuse, KV-cache dtypes, disaggregated serving, multimodality and speculative decoding) and can then tune its serving performance. Use when the user asks to onboard a model with the modeling v2 agent flow, gives a goal naming a checkpoint, features, parallel configurations (TP, TEP, DEP or ADP) and a workload, or says "run trtllm-modeling-v2". Works from Claude Code and from Codex, including a Codex `/goal`. |
| 4 | +license: Apache-2.0 |
| 5 | +metadata: |
| 6 | + author: NVIDIA Corporation |
| 7 | +--- |
| 8 | + |
| 9 | +# Operating the trtllm-modeling-v2 workflow |
| 10 | + |
| 11 | +The workflow is a Python orchestrator: you prepare its inputs, start it, watch |
| 12 | +it and steer it. The agents it starts write the code, and Python checks decide |
| 13 | +when a feature is done. Do not build the model by hand beside it, and never |
| 14 | +declare a check passed yourself. |
| 15 | + |
| 16 | +The package is `agent-flow/agent_flow/workflows/trtllm_modeling_v2/` (below: |
| 17 | +`PKG`). Read, as you need them: |
| 18 | + |
| 19 | +- `PKG/README.md`: what a run does, every flag and subcommand. |
| 20 | +- `PKG/INTAKE.md`: §2 the fast path and its goal words, §3 `site.yaml`, |
| 21 | + §4–5 the request and its features, §9 the intake commands. |
| 22 | +- `PKG/docs/user-guide.md`: prerequisites (§2), the spec (§5), compute |
| 23 | + sessions (§6), starting (§8), watching (§9), steering (§10), tune (§11). |
| 24 | +- `PKG/task.example.yaml`: every key of the resolved spec. |
| 25 | +- `PKG/TROUBLESHOOTING.md`: search it for the verbatim error before you |
| 26 | + debug the model; environment failures read as model failures. |
| 27 | + |
| 28 | +## 1. Checkouts and install |
| 29 | + |
| 30 | +- **The target checkout** is the TensorRT-LLM checkout the run builds the |
| 31 | + model in, on a branch of its own: from `main` when asked for a fresh one |
| 32 | + (`git -C <target> fetch origin && git -C <target> switch -c <branch> origin/main`). |
| 33 | + It needs a native build and the interpreter that imports it |
| 34 | + (`environment.python`, inside the container on a cluster), and a git |
| 35 | + identity (`user.name`, `user.email`). User guide §2. |
| 36 | +- **The workflow's code**: when the target checkout does not have `PKG` yet, |
| 37 | + take it from the branch or pull request the user names, in a worktree of its |
| 38 | + own so the target branch stays clean: |
| 39 | + `git -C <target> fetch origin pull/<N>/head:mv2-flow && git -C <target> worktree add <flow> mv2-flow`. |
| 40 | +- **Install** it into a venv of its own (Python 3.11 or later), and run every |
| 41 | + command below with that venv's `trtllm-modeling-v2`: |
| 42 | + `python3 -m venv <venv> && <venv>/bin/pip install -e <flow>/agent-flow`. |
| 43 | + |
| 44 | +## 2. From the request to a spec |
| 45 | + |
| 46 | +Write `site.yaml` once per cluster (INTAKE §3): `trtllm_repo_path` (the |
| 47 | +target checkout); `environment` (`session_spool` on a Slurm cluster: |
| 48 | +`partition`, `account`, `container_image` — a `.sqsh` on shared storage |
| 49 | +mounts in seconds —, `mounts`, `python` inside the container, `sessions`); |
| 50 | +`device: {arch, gpus_per_node, nodes}` (`arch: sm_103` for GB300, `sm_100` |
| 51 | +for B200); `agents` (section 3); and `defaults`. |
| 52 | + |
| 53 | +Turn the request into a goal line (INTAKE §2): |
| 54 | + |
| 55 | +- **Members**: `tp<N>`, `tep<N>`, `dep<N>`; attention data parallel (ADP) is |
| 56 | + `dep`. |
| 57 | +- **Features** by their INTAKE §5 names (`disagg`, `multimodal`, `kv_quant`, |
| 58 | + `mtp`, ...). `<feature>@dep4` gates a feature on those members only; the |
| 59 | + others are then proven on the core alone. A checkpoint's vision tower is |
| 60 | + built unless the goal says `-multimodal`. |
| 61 | +- **Core switches** `cuda_graph`, `overlap_scheduler`, `chunked_prefill` and |
| 62 | + `prefix_reuse` (block reuse) are on unless the goal says `-<switch>`. |
| 63 | +- **Workload**: `isl=8192 osl=1024` for 8K/1K. `tune` tunes the target once |
| 64 | + it is delivered. |
| 65 | +- **KV-cache dtype**: one is `kv=fp8`. Several (bf16, fp8, nvfp4) are |
| 66 | + `variants` of the unit (`PKG/task.example.yaml`), which intake does not |
| 67 | + carry: add them to the compiled `task.resolved.yaml`. |
| 68 | + |
| 69 | +Derive without starting, read the report, then check and compile: |
| 70 | + |
| 71 | +```bash |
| 72 | +trtllm-modeling-v2 intake derive --checkpoint <ckpt> --site site.yaml --goal "<goal>" -o <W>/task.intake.yaml |
| 73 | +# read <W>/intake.report.md: what was read, defaulted, and is still open |
| 74 | +trtllm-modeling-v2 intake check <W>/task.intake.yaml |
| 75 | +trtllm-modeling-v2 intake compile <W>/task.intake.yaml --site site.yaml -o <W>/task.resolved.yaml |
| 76 | +# add what intake does not carry (variants, perf.campaign) to task.resolved.yaml |
| 77 | +``` |
| 78 | + |
| 79 | +**Accuracy anchors.** A unit with no measured row yet runs with |
| 80 | +`accuracy.anchor: pending`: the logit gate is then the oracle, so make its |
| 81 | +reference once from the reference implementation (user guide §2). Add the row |
| 82 | +later from a measurement or from the user. Never invent one. |
| 83 | + |
| 84 | +Keep the workspace `<W>` on the shared filesystem, outside every checkout. |
| 85 | + |
| 86 | +## 3. Which agent runs the turns |
| 87 | + |
| 88 | +The orchestrator starts every role turn itself, as an agent session. |
| 89 | + |
| 90 | +- **From Claude Code** the roles run on Claude Code. Start the orchestrator |
| 91 | + from a plain terminal or tmux, not from inside the Claude Code session: a |
| 92 | + turn started there inherits its sandbox (user guide §8). |
| 93 | +- **From Codex**, an interactive session or a `/goal`, the roles run on |
| 94 | + Codex: on the model your Codex is set to, with the newer of your `codex` and |
| 95 | + the pinned runtime. Run that Codex session with full access |
| 96 | + (`codex --dangerously-bypass-approvals-and-sandbox`, or |
| 97 | + `sandbox_mode = "danger-full-access"`): the orchestrator, its gates and its |
| 98 | + turns need Slurm, the shared filesystem and the network outside Codex's |
| 99 | + sandbox. Put `agents: {defaults: {backend: codex}}` in `site.yaml` as well, |
| 100 | + so a relaunch from anywhere keeps it, and add `model:` to pin one your login |
| 101 | + offers. |
| 102 | +- **Mixed**: route single roles under `agents.roles` (`PKG/task.example.yaml`). |
| 103 | + |
| 104 | +The dry run's agent probe proves each configured agent can take a turn on this |
| 105 | +host before anything is spent; when it fails, its message says what to fix. |
| 106 | + |
| 107 | +## 4. Compute sessions |
| 108 | + |
| 109 | +With `--manage-sessions` the run submits, renews and releases its sessions |
| 110 | +itself, from a host with `sbatch`, `squeue` and `scancel`. Otherwise render |
| 111 | +and submit them yourself (user guide §6): |
| 112 | +`trtllm-modeling-v2 session render|submission|status --task <W>/task.resolved.yaml --name <session>`. |
| 113 | + |
| 114 | +## 5. Start, watch, steer |
| 115 | + |
| 116 | +```bash |
| 117 | +trtllm-modeling-v2 --task <W>/task.resolved.yaml --workspace <W> --dry-run # preflight; read its notes |
| 118 | +setsid nohup trtllm-modeling-v2 --task <W>/task.resolved.yaml --workspace <W> [--manage-sessions] \ |
| 119 | + > <W>.log 2>&1 < /dev/null & # outlives this shell |
| 120 | +trtllm-modeling-v2 driver --workspace <W> status # turns, orchestrator, sessions |
| 121 | +trtllm-modeling-v2 driver --workspace <W> wait --sessions-only # blocks until a session or the orchestrator needs you |
| 122 | +trtllm-modeling-v2 feedback --workspace <W> "message" # reaches the planner's next turn; relaxes no check |
| 123 | +``` |
| 124 | + |
| 125 | +- `<W>/journal.md` is the run's account of itself; `<W>.log` is the process |
| 126 | + output. |
| 127 | +- Re-running the same start command resumes the run; a second orchestrator on |
| 128 | + the workspace is refused. |
| 129 | +- `ORCHESTRATOR_EXITED` from `driver wait`: read `<W>.log`, and relaunch |
| 130 | + unless the run closed. |
| 131 | +- The run closes `delivered` (branch `<branch>-correct`, `cells.json`, |
| 132 | + `metrics.json`) or `tuned`. Tune results are listed in |
| 133 | + `<W>/UNMERGED_OPTIMIZATIONS.md` for a human to merge. |
0 commit comments