[Wanli-Jiang]

!! Let's use #20907 which is more relied on LLM ability to figure the code organizations.

@coderabbitai summary

Description

What trtllm_modeling_v2 is

trtllm_modeling_v2 is an agent-flow workflow that takes one Hugging Face checkpoint to a gated ModelingV2 target, then makes the target fast without making it less correct. A target is a flat, self-contained forward at tensorrt_llm/_torch/_experimental/modeling_v2/models/<family>/<checkpoint>__<arch>__<parallel>/, built from catalog entries and the target's own ops/. It is trusted because it passes accuracy gates, not because it shares abstractions with other models. The modeling_v2 catalog, targets and router are already in tree (#19977); this PR adds the agents that build them.

A run has these steps:

1. Intake. Either trtllm-modeling-v2 --checkpoint <hf_dir> --site site.yaml --goal "<features, members, workload>" derives the spec from the checkpoint's config, the site's defaults and a one-line goal, or intake check|confirm|compile turns a user-facing task.yaml into it. Intake checks are the first part of phase 0.
2. Preflight and phase 0 (Python, no agent turn):
   • checks the checkpoint, the host and the launch substrate;
   • runs a stock-path cold start beside the first turns, and measures the serving geometry (pools, pages, chunk splits, capture list) before any kernel task;
   • surveys the catalog surface, and warns of known engine defects the target tree lacks the fix for;
   • writes contract.yaml, the list of every check the run owes: boot, accuracy, witnesses, acceptance, logits, serve and envelope, at every parallel member.
3. Planned run.
   • A planner writes a task graph with these task kinds: survey, kernel, assemble, feature, integrate, investigate, review, check, tune. A check task runs contract checks with no agent turn; depends_on_soft lets a task start before its dependency lands.
   • The orchestrator starts tasks as capacity and their files allow. Each task runs in its own sparse worktree and lands by rebase and fast-forward.
   • The orchestrator runs the checks each task claims. A worker reports that it finished, never that a check passed.
4. Milestones. BRINGUP DELIVERED <sha> marks the core passing at tp1. The first tree on which every required check holds is recorded as <branch>-correct and CORRECTNESS DELIVERED <sha>. Neither waits for performance work.
5. Tune, a task that joins once the core is correct:
   • ground truth: the baseline, the in-tree reference and a noise floor;
   • a config ladder;
   • a campaign of agent-written candidates, each measured against a same-host control and kept only if it beats the noise floor and passes the contract's own accuracy gate.

An improver runs beside the run. It is woken by what costs the run time, and turns failures into scoped lessons, patch drafts for workflow defects, and a registry of known engine defects, all kept across runs.

The documentation set is in docs/: a design doc, a user guide, a developer guide, an end-to-end worked example and the measured results.

What changed since the previous revision

The previous revision ran a fixed staircase: plan, then assemble one unit at a time, then gate, then a remediation cycle run by an auditor, then deliver. This revision replaces that with a contract plus a task graph, and its history is rewritten as the 13 commits below. Two from-scratch onboardings of one real checkpoint then hardened it:

• Run 1 found the defects while it ran. They were fixed live: 14 orchestrator restarts and about 60 flow changes.
• Run 2 ran the fixed flow, pinned. It needed no restart and no flow change.
• What run 2 still measured wrong or slow is addressed in the last two rounds of changes, listed below. They are tested on CPU only; the next from-scratch run validates them on GPU.
• Run 3 (from scratch, flow pinned) measured the faster start: the first code task at 28 min (run 2: 2 h 15), the kernel fan-out at 39 min (run 2: 3 h 09) and the core's assembly dispatched at 3 h 55 (run 2: 5 h 28). It was then stopped by an operator-side outage, and its time-to-full-support analysis drives the round below.

Added:

• Intake, and a contract derived from the spec.
• The planner, plus an advisory graph critic that holds back only the tasks a graph adds.
• Parallel tasks in sparse worktrees. A task's receipts and the measured runtime facts reach the tasks that depend on it.
• Verdicts fingerprinted by the tree the gates read:
  • a pass is shared between checks that measure one engine;
  • a stale verdict runs again, riskiest first.
• Engine attestation: every gate records the engine it actually ran, plus its pools and free memory.
• A logit gate against the HF reference, and a trtllm-serve gate.
• Session-aware scheduling of GPU jobs across Slurm sessions.
• The external-transport driver with reload.
• The correctness milestone, and a tune campaign that runs beside the build.

Added in the last rounds (CPU-tested, to be validated by the next run):

• Checks measure a read-only snapshot of one commit; an equivalence witness proves it can see lost state; block reuse must be shown to hit.
• Faster start: the survey starts at minute 0, the survey's queue rows become kernel tasks without a planner turn, the critic holds nothing, a cap on concurrent agents, and intake is part of phase 0 (with a fast path).
• Fewer and cheaper checks: one engine serves several checks of one (unit, member, commit) (gate bundles); with the logit gate on, the accuracy benchmark runs at tp1, the widest and attention-DP members; boot compares greedy tokens with the HF reference; a certification whose cells are already covered closes without a turn; regression reviews are batched and use a test-result cache.
• Less serialization: certifications on one shared catalog entry land as local variants in parallel and one promote task merges them; check tasks and soft dependencies.
• The disaggregated witness ships its own leg B, so no per-model script is needed.
• The improver drafts patches and keeps known engine defects.

Added after run 3 (CPU-tested), each aimed at the time to full support:

• At most --max-parallel-tasks agents (default 8) work at once, the planner, critic and improver included; task turns leave one slot for the planner.
• Kernel rows start in natural id order and the core's rows first; a failed receipt whose entry landed on the target no longer holds its dependents' hard edges.
• No planner turn for an event the graph already answers, and no improver turn for a wake an open flow issue already lists.
• The planner links the core to kernel rows by soft edges, starts widths as check tasks, answers failures first in a small revision and keeps its first turn to what can start now; a smith hands back a failed receipt as soon as the defect is in shared code; workers run only their own tests and the cross-model regression runs once in the review task.
• TMPDIR, a multimodal processor's dependencies and the large pytest marker are checked before sessions or agents start; every gate gets a per-run Triton cache.

Added after a design review of run 3 (CPU-tested), each fixing a class of defect rather than one instance:

• One definition of done. Full support (every required check, the delivery files on disk, every shared change reviewed, no local variant waiting) delivers the run and stops a running tune campaign; tune is background work on idle agent slots and nothing depends on or integrates its branch.
• One capacity model. Agent slots are granted by priority (planner, critical, required, side, background), one slot stays free for the planner.
• Fault containment. A planner turn that raises is asked again on the same events, an unreadable graph is refused and sent back, a check whose session dies leaves its task landed and runs again, and no finished result is lost.
• Continuous verification. A member's delivery tier is rehearsed once its other checks hold, a landing is re-checked beside the work still running, the orchestrator adds the task that writes the delivery files, and promotion is per entry.
• What "fully supported" means is written down. Preflight expands the request into a feature x member matrix (coverage.yaml, and the delivery report): each cell certified, sampled or not covered with the reason, and an uncovered cell stops the run unless task.yaml accepts it.
• Each check proves what it claims. Per feature, the evidence a check must observe and the oracle it is judged by; every rank's worker reports its own engine, so a multi-rank member is never attested from the request; a chunked variant must serve a prompt it splits; each speculation depth must beat the one below where no stock speculator exists; serve answers are held to the boot workload's keywords; a multimodal unit counts as judged only by a check that sends images.
• A verdict is about every input it read. Each check carries a digest of the spec as it applies to its unit and of the files outside the checkout; verdicts on other inputs are stale, options are compared by content, and acceptance checks share verdicts. Certifications close and delivery holds only on recorded test passes.
• Engine defects become workflow knowledge. A registry keyed by what triggers a defect (where, the parameter regime, the detecting test, the fix or workaround): rows in a known defect's regime are marked when the survey lands, and a smith's structured engine_defect receipt is registered and gets its fix task automatically. knowledge: measurement leaves it out of a calibration run.
• Geometry from the target. Once the core boots at a width, the target's own geometry there is probed, and what no certification saw becomes one re-certification batch; the kernel skills take their scope from the dispatch.
• One decision table renders every role's "Ending your turn" section and is the enum the run validates and the set the orchestrator finishes on.
• Whole runs on the CPU. A simulation drives the real orchestrator from the first graph to delivery through the tail no GPU run has reached.

Added after runs 4 and 5 (CPU-tested), aimed at quick onboarding of any model. Two more from-scratch runs on the design-review revision delivered bring-up in 4.6 h and 4.1 h, and neither reached correctness. What held them back was generic:

• A verdict stands while what it read is unchanged. Every process of a check records the tree's Python modules it imported; a verdict holds on a later commit while those files, every non-Python file, the native libraries and the anchor entries are unchanged. Previously any landing made every verdict stale (one late catalog merge made 194 of 195 stale).
• Milestones name the commit the checks measured, wherever the checkout has moved since; the run's clock starts when it goes past preflight, not at a dry run.
• One setting, every launcher. A setting both the quickstart gates and the options-file launchers take (KV fraction and dtypes, tokens per block, max_num_tokens, trust_remote_code) is stated once and handed to the other; two values are refused at preflight; trust_remote_code follows the checkpoint's auto_map; a one-rank command runs its worker in process.
• No anchor, no fake anchor. accuracy.anchor: pending makes the logit gate the oracle and keeps delivery incomplete until a written anchor exists; an anchor of 1 or less is refused.
• Promotes early and in parallel: a promote declares only the files it rewrites, cells-only variants are due at once, a due promote holds the checks that have not started; the survey writes one queue row per shared entry, its cells spanning the served envelope; what a landed entry or unit leaves uncertified reaches the planner.
• Fewer wasted turns and slots. An orchestrator note names what answers it, so a revision that already holds it wakes no planner turn, and a passing check task wakes none; the planner's reserved slot takes nothing from the tasks; critic and improver turns use idle slots; a ready task's wait ages its rank; a production run's improver turns once, at the close.
• GPUs follow the work. A queued gate moves to a session that would start it now (an atomic rename, so it never runs twice); a session near its end judges a job by how long such jobs ran, not its worst-case wall limit; a dev session lends free devices beside the gates it already took.
• Robust reloads and builds. A turn a stopped process left is adopted only for its own work; unread improver and planner proposals are merged or set aside, not deleted; tasks the orchestrator adds are wired before the regression review they need; an engine-defect fix task carries the receipt's whole proposal; a native build installs into a venv of its own, never the run's.

Removed:

• the auditor and its remediation cycle;
• the evidence ledger and its reports (replaced by the contract and its verdicts);
• the premerge pack and tool-enforced embargo hooks (the embargo is now a rule in every role's prompt);
• --overlap-units (tasks run in parallel by default);
• the scheduler-equivalence and launch gates;
• the eplb/dwdp features;
• the slurm, slurm_broker and per-workload launch paths (session spools only);
• the staged agent home.

Validation on a real model

Both onboarding runs used the same unreleased checkpoint, which has no in-tree implementation. Setup:

• Hardware and workload: GB300; ISL 8192 / OSL 1024.
• Parallel members: tp1 tp2 tep2 dep2 tp4 tep4 dep4.
• Feature units: core, MTP (depth 1–7), disaggregated serving and multimodal, plus chunked prefill, block reuse and cache-dtype variants.

Accuracy anchors were measured with the HF reference.

Onboarding, run 2 against run 1. Both are timed from the start of the run, over the same first 11.9 h:

	run 1 (flow fixed while it ran)	run 2 (flow pinned)
core boot + accuracy pass at tp1	9 h 38	6 h 45
all four units (core, mtp, disagg, multimodal) integrated at tp1	built width-first; all widths at 18 h 56	10 h 51
contract checks passing at 11.9 h	10	84
workers in flight, mean / peak	2.46 / 6	4.97 / 10 (the cap)
GPU busy	20.7 GPU-h	73.8 GPU-h
verdicts per check (a check run more than once on one tree shows here)	2.83	1.00
contract FAILs	7	0
orchestrator restarts / flow changes during the run	6 / 9	0 / 0

Accuracy: in run 2, the core's accuracy gate passed at all seven members, and MTP acceptance passed at every depth it was gated at.

Neither run reached the correctness milestone. Both were stopped by the operator:

• Run 1 was stopped after about 46 h. The final tier had not run on its last head.
• Run 2 was stopped at 12 h 42 so that its findings could go into the flow. 93 of 139 contract checks were passing; the multimodal and disagg wide members and the final tier were still running.

Run 2's findings, addressed in this revision but not yet re-measured on a run:

• checks ran on the live checkout;
• the disagg witness used filler prompts that were blind to lost state;
• the attention-DP hand-over only ever reached rank 0;
• block reuse was never shown to hit;
• the first code task started 2 h 15 after the run opened;
• tune held gate GPUs it never used.

Tune, on run 1's target. The table compares 21 headline points (7 members × 3 metrics) against the tune baseline.

tree	geomean speed-up	median % of SOL	points ≥ 50% SOL
tune baseline	1.00×	26%	2 / 21
campaign's best kept tree	1.84×	53%	11 / 21
+ two allreduce changes landed after it	1.97×	53%	13 / 21
+ member-gated variants (measured, not yet landed when the run stopped)	2.07×	56%	14 / 21

• Measurement caveats:
  • the last two rows were measured on two other hosts than the first two;
  • SOL is a hand-reduced estimate.
• Where the gain came from: fusions, kernel sourcing, and one-shot and MNNVL allreduce.
• Accuracy of the variant tree: the accuracy gate passed at all seven members, and so did MTP acceptance.
• TTFT got worse at two high-concurrency points: TTFT is not part of the keep rule, a gap listed under limitations.

The tune code path was also validated earlier on a Qwen3.8-Flash-Next-NVFP4 target: 1.11× and 1.28× over the in-tree model with matched kernel switches. The staircase-era onboardings of MiMo-V2.6 and DeepSeek-V4.1-Flash-NVFP4 are condensed in LESSONS.md.

The runs also found engine defects outside modeling_v2. Each goes in its own PR, not in this one:

• FLA gated delta rule with tied gates;
• 64-bit conv-state offsets in causalConv1d;
• thop attention with a bf16 output over an NVFP4 KV cache;
• a routing hook for a target's input processor;
• disaggregated multimodal hash relay;
• MoE fake-op registrations;
• the KV-cache estimate for MQA under TP.

What changed (by area)

#	Area	Main points
1	Spec and intake	The task.yaml schema, intake check|confirm|compile with a site.yaml, and the fast path that derives the spec from the checkpoint, the site and a goal. Configuration variants, and members of different widths. A capture-list agreement check and a KV-cache configuration check for hybrid models (reuse without a snapshot policy).
2	Preflight and contract	Pure-Python preflight and phase 0: intake checks, the substrate gate beside the first turns (a resumed run stands on a recent pass), the geometry probe, and the known-defect warning. For in-process runs, a probe that proves every agent can run Bash. contract.yaml lists every check owed; the accuracy benchmark runs at tp1, the widest and attention-DP members when the logit gate covers the rest.
3	Planned run	run/graph.py decides which tasks may run and when (file touches, exclusive, nice_to_have, attempts, check tasks, soft dependencies, batched regression reviews, promote). The planner runs on its own thread with an event window. Receipts and runtime facts are handed to dependent tasks. An infrastructure failure is retried once. A relaunched orchestrator adopts answered turns, and driver reload drains it and re-executes it on new flow code.
4	Worktrees and integration	Sparse task worktrees that start in seconds and write no repository config. Integration is rebase plus fast-forward, so the checkout carries no merge commits. A conflict goes back to the task's worker, up to twice.
5	Checks	Gates: boot (matched on generated text), accuracy (anchor − k·SE), witnesses (disagg equivalence, kv_quant, host offload, bounded replay), acceptance (paired, ≥ 32 natural-text prompts), logits (per-position log-probs against the HF reference), serve (trtllm-serve), envelope. Every check runs with modeling_v2=require and attests the engine it ran. Verdicts are fingerprinted by the tree they read and shared when one engine is measured. Checks are admitted by the sessions' free devices. Gate bundles share one engine across checks; boot compares greedy tokens with the reference; the disaggregated witness's leg B ships with the flow; a test-result cache runs only the tests a change affects.
6	Execution substrate	A spool scheduler per Slurm session: device slots; a device is quarantined until its memory is released; jobs are routed to a session that can run them now and borrow idle devices. Sessions are declared in task.yaml and handed over by allocation, with a watchdog. The external-transport driver (driver wait|claim|prompt|status|reload). In-process turns leave their files and keep their role's time bound.
7	Tune	The campaign runs as a task beside the build, with SOL targets. The keep rule is judged against the tree's own noise and only at the members a change touches; the keep gate is the target suite plus the contract's accuracy gate. A loss split by member yields a member-gated variant. Stacked evaluation, perf.campaign.max_hours, correctness delivered beside a running campaign. Plus bench.py, nsys_summary.py, microbench.py, and the perf CLI.
8	Lessons and improver	A scoped lesson store (all, role:, host:, family:). A workflow_defect lesson files a flow issue with an optional validated patch draft, an engine_defect lesson names its upstream fix, and a lesson is retired once the commit that fixed it is in the running flow. The improver runs beside the run, woken by failed receipts, repeated certifications, stalled ready tasks, no-op planner turns, VOID verdicts and reported flow issues.
9	modeling_v2 as merged (#19977)	Catalog entries are OpWrapper classes. CELLS and ARCHS are the executable contract that the survey reads and delivery checks. Delivery checks _select_target, the W-row weight table and routing. Gates pass modeling_v2 through the LLM API and set TRTLLM_MODELING_V2_VALIDATE=1. FlashInfer is a sourcing tier.
10	Docs	docs/ (design, user guide, developer guide, examples, results), README, INTAKE, BUILD_AND_TEST, TROUBLESHOOTING and LESSONS.md. A short "environment failures that read as model failures" table in AGENTS.md.

Outside agent-flow/agent_flow/workflows/trtllm_modeling_v2/ and its tests, the PR touches only:

• agent-flow/agent_flow/backends/ and agent-flow/tests/test_backends.py. In-process turns launch the agent CLI named by the backend's CLI-path variable, or a newer CLI on PATH, instead of only the SDK's bundled binary, which an organization's minimum-version policy can refuse.
• agent-flow/pyproject.toml: the trtllm-modeling-v2 entry point, package data, and an E501 per-file ignore for prompt tables.
• agent-flow/README.md: one sentence.
• AGENTS.md: one table.

There are no changes under tensorrt_llm/, cpp/ or tests/, and no new dependencies.

Commits

The PR is 13 commits, one per layer, each carrying its own tests, so it can be read in order:

1. agent_flow: the CLI agent backend (the only change to agent-flow's core).
2. The task spec, intake and configuration variants.
3. Compute sessions, the spool scheduler and command execution.
4. Launch environment, git, commits and the external transport.
5. The survey, the vocabulary, the runtime map and delivery checks.
6. The contract's gates, witnesses and their harness.
7. The roles, their prompts and the lessons store.
8. The skills the roles load.
9. The contract, the task graph, integration and run records.
10. Preflight, the orchestrator loop and the workflow.
11. The tune stage.
12. The CLI and package entry point.
13. Docs.

Modules refer to later layers, so only the last commit is complete. The tests below ran on it.

How to run

pip install -e agent-flow                       # provides the trtllm-modeling-v2 CLI

# 0. The fast path: derive the spec and start
trtllm-modeling-v2 --checkpoint /path/to/hf_checkpoint --site site.yaml --workspace ws/ \
    --goal "mtp,disagg tp1,tep4 isl=8192 osl=1024" --confirm <you>

#    or a hand-written spec (INTAKE.md); task.example.yaml documents every key
trtllm-modeling-v2 intake check task.yaml
trtllm-modeling-v2 intake confirm task.yaml --by <you>

# 1. Cheapest check first: preflight, contract and substrate probe; no planner turn
trtllm-modeling-v2 --task task.yaml --site site.yaml --workspace ws/ --dry-run
trtllm-modeling-v2 --task task.yaml --site site.yaml --workspace ws/   # resumable: re-run the same command

# 2. Where the orchestrator cannot run an agent itself (sandbox, rotating tokens)
trtllm-modeling-v2 --task task.yaml --workspace ws/ --agent-transport external
trtllm-modeling-v2 driver --workspace ws/ wait           # NEW <id> | ORCHESTRATOR_EXITED | ...
trtllm-modeling-v2 driver --workspace ws/ prompt <id> --claim
trtllm-modeling-v2 driver --workspace ws/ reload         # load changed flow code into a running run

# 3. One gate on its own, exactly as the orchestrator runs it
trtllm-modeling-v2 gate <name> --task task.yaml --unit <unit> --member <member> --tree <checkout>

# 4. Performance on a delivered target
trtllm-modeling-v2 tune --task task.yaml --workspace ws/

Test Coverage

All tests run on CPU and need no GPU, model weights or cluster. Pass PYTHONPATH=agent-flow when an editable install points at another checkout.

• pytest agent-flow/tests/workflows/trtllm_modeling_v2 on this revision: 2936 passed, 2 skipped. The previous revision's full pytest agent-flow/tests was 4479 passed, 5 skipped. The trtllm_modeling_v2 suites include:
  • the graph, planner, integration and receipt suites;
  • the gate suites, with engine attestation, witness ablation and reuse attestation;
  • the substrate suites, with fault injection;
  • the tune suites;
  • the phase-0, check-task, soft-dependency, promote, gate-bundle, cheaper-oracle, leg-B, test-cache and improver suites added in the last round;
  • the design-review round's suites: capacity, fault containment, continuous verification, requirements, observables, verdict inputs, engine defects, target geometry, the decision table, and test_flow_simulation (whole runs on the CPU, first graph to delivery);
  • the last round's suites: verdict read sets (test_verdict_reads), launch settings (test_launch_settings), turn adoption by work, pending anchors, promote timing, gate relocation and expected run times;
  • test_delivery against the in-tree gpt_oss and DeepSeek-V3 targets;
  • test_package_hygiene, which keeps checkpoint specifics and personal paths out of the package.
• Lint: the repo's .pre-commit-config.yaml passes on every file this PR changes.

GPU validation is the two onboarding runs and the tune campaign described above.

Known limitations and follow-ups

• Correctness has not yet been delivered end to end on this revision. The next from-scratch run is planned with these targets: correctness ≤ 16 h, ≤ 3 late re-certifications.
• The round after runs 4 and 5 is CPU-tested only. Verdict read sets depend on every gate process importing the rank hook; a check whose processes do not report falls back to the whole-tree fingerprint. No class-A (all ops already in the catalog) model has been run yet.
• The design-review round is CPU-tested only. Per-rank engine records come from a sitecustomize hook on the gates' PYTHONPATH; it is exercised against a fake engine in a real subprocess, not yet under mpirun on GPUs. No check yet proves image inputs across the disaggregated hand-over: the requirements matrix shows it as an uncovered row a run must accept.
• The last rounds are CPU-tested only. Gate bundles, the geometry probe, the flow's own disaggregated leg B, the test-result cache, promote and the improver's patch drafts run against fakes in the tests; the next from-scratch run is their first GPU validation.
• The faster start is measured, the rest of the run is not yet. Run 3 started its first code task at 28 min, but stopped before the core's tp1 certification; the round added after it is CPU-tested only.
• The intake fast path does not yet express everything a full run needs (variants, per-depth acceptance floors, envelope and perf campaign settings); such runs still use a hand-written task.yaml.
• Performance is measured, not gated, and TTFT is not part of the keep rule.
• The reference model's switches are not tuned. The ladder never tries engine switches for the in-tree reference.
• The embargo is a prompt rule, not a tool-enforced one.
• No DSpark branch in main's quickstart. A dspark spec stops at preflight with an explicit message.

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

[litaotju]

Thanks for putting all these together.

1. Lets try to rename staircase to "trtllm_modeling_v2"?

2. One open question. Shall each feature have its own skill or it should be part of the main skill with reference doc on how each feature shall be implemented?

[Wanli-Jiang]

updated codes and comments in #20145 (comment)

[WeiHaocheng]

I am still understanding and reviewing the code in the workflow section.

Regarding the "skills" section, I do not recommend incorporating these skills into the repo. The previous experience of the modeling agent tells us that, first, detailed skills may limit the model's performance. Second, the large amount of information carried in the skills is difficult to maintain, and errors and outdated information will become difficult to manage. We can first place them in an internal small repo. If there are clear AB tests proving that this is effective, then we can incorporate them into the repo.

[Wanli-Jiang]

Thanks both for the reviews. I've force-pushed an updated branch: it is rebased onto main now that #19977 has merged (so only this PR's own commits remain), aligned with modeling_v2 as merged, and it adds the tune stage. The description is updated to match.

@litaotju

1. Rename: done. The workflow is now trtllm_modeling_v2: package agent_flow.workflows.trtllm_modeling_v2, CLI trtllm-modeling-v2, TrtllmModelingV2Workflow, and skills trtllm-modeling-v2-* (following the repo's trtllm- skill prefix). The workflow's own env vars became MV2FLOW_* rather than TRTLLM_MODELING_V2_*, so they can't be mistaken for the engine's TRTLLM_MODELING_V2 / _VALIDATE / _VALUE_DUMP switches. In the code, "staircase" now appears only in two quoted historical error messages.
2. One skill per feature, or one main skill with per-feature references? Today it is one core skill plus one skill per feature. The reason: a run assembles one unit per feature, and each assembler turn is told to load the core skill plus only its own unit's feature skill (FEATURE_SKILLS in task_schema.py), so a turn never carries the other features' text. A main skill whose references are named per feature would keep that property as long as the turn prompt names the reference to read, so I'm happy to restructure it that way if you prefer it; it would also shrink the surface @WeiHaocheng is concerned about (below).

@WeiHaocheng

Both concerns are fair, and this week gave a concrete example of the second: aligning with modeling_v2 as merged meant rewriting 32 skill documents. Some context on how these skills differ from .claude/skills, then a proposal.

• They are inputs of this workflow rather than general-purpose skills: they are versioned with the prompts and gates that reference them, packaged with the workflow, and copied into each run's workspace, so a run never depends on whatever a checkout's .claude/skills happens to contain.
• Part of their correctness is tested: test_skills checks that every repo path a skill names exists, and test_prompts now refuses the pre-merge modeling_v2 names. That catches path-level drift, not semantic drift, which is exactly the staleness you describe.

Proposal, and I'd like your view before I change the code:

• (a) Slim in tree. Fold the per-feature skills into the core skill's references (answering @litaotju's question 2 at the same time) and cut the content to the rules a gate or delivery check can't enforce on its own. The volume drops and what remains stays versioned with the workflow.
• (b) Out of tree. Add a skills_dir option to task.yaml, ship only the minimal set the prompts require, and keep the full skills in an internal repo until an A/B (the same onboarding with and without them) shows they help.

I lean towards (a) for this PR plus the A/B as a follow-up, but would go with (b) if you'd rather keep the detailed text out of the repo until there is data. Which do you prefer?

[Wanli-Jiang]

Another agent flow is #20792