Skip to content

SPEC.md

Latest commit

 

History

History
529 lines (437 loc) · 39.4 KB

SPEC.md

File metadata and controls

529 lines (437 loc) · 39.4 KB
Raw

TRINITY Replication — Canonical Implementation Spec (SPEC.md)

Generated from a multi-agent deep read of the paper, then corrected by an adversarial review and grounded against the real Qwen3-0.6B config and a live OpenRouter probe.

Section 0 below is authoritative. Where any number later in this document disagrees with Section 0, Section 0 wins (it folds in the review corrections + verified facts).

0. VERIFIED FACTS & CORRECTIONS (authoritative — override anything below)

0.1 Verified against ground truth (not the paper's prose)

  • Qwen3-0.6B real config (huggingface.co/Qwen/Qwen3-0.6B/config.json): hidden_size = 1024 → d_h = 1024 CONFIRMED; num_hidden_layers = 28 → second-to-last layer = index 26; num_attention_heads = 16, num_key_value_heads = 8 (GQA), head_dim = 128; intermediate_size = 3072 (SwiGLU: gate/up/down); tie_word_embeddings = True; bfloat16. Per-layer linear matrices and their SVD singular-value counts (= min dim): q_proj 1024×2048→1024, k_proj 1024×1024→1024, v_proj 1024×1024→1024, o_proj 2048×1024→1024, gate_proj 1024×3072→1024, up_proj 1024×3072→1024, down_proj 3072×1024→1024. All 7 matrices give 1024 SVs each.
  • OpenRouter pool verified LIVE (HTTP 200, real completions): qwen3.5-35b-a3b, gemini-3.1-flash-lite, deepseek-v4-flash all exist and answer.
  • Remote box: 8× H200 NVL (143 GB), GPU index 5 only, /mnt/data 3.2 TB free.

0.2 Corrected numbers (the review caught arithmetic errors below)

Quantity WRONG (in body) CORRECT (use this) Why
Pool size L — 3 our setup
Head output n_a = L+3 — 6 3 agents + 3 roles
Linear head params — 6,144 = 6×1024 correct
SVF scales "9,216 (Q/K/V/O+MLP)" 7,168 = 7×1024 [OUR CHOICE] paper's 9,216=9×1024 does NOT map onto Qwen3's 7 linear matrices; SVF-ing all 7 of layer 26 = 7,168. Verify empirically (S2).
Total trainable n 15,360 13,312 = 6,144 + 7,168 head + SVF
CMA population λ "34" 33 = ⌈4 + 3·ln(13312)⌉ = ⌈32.49⌉ review caught 34 is wrong (also 33 for n=15,360)
Parents μ 17 16 = ⌊33/2⌋ follows λ
Budget B_env "34,560" 31,680 = 16·33·60 16·34·60≠34,560 anyway; product error
REINFORCE batch = m·λ "544" 528 = 16·33 followed wrong λ

0.3 Gaps the review exposed — decisions now fixed (were silently assumed)

  1. SVF matrix set & count: apply SVF to all 7 linear matrices of layer 26 → 7,168 learnable singular-value scales, all initialized to 1.0 (identity ⇒ unmodified SLM at start). The smoke test S2 must print the actual count and assert it equals what we pack into θ; do not trust 7,168 blindly until S2 confirms it on the loaded checkpoint.
  2. Hidden-state norm / σ₀ coupling: raw bf16 penultimate hidden states can have large norm, so σ₀=0.1 on a W=0 start may saturate softmax. OUR CHOICE: L2-normalize h (h ← h/‖h‖) before the head, making logit scale ‖h‖-independent and σ₀ well-behaved. S1 prints ‖h‖ to confirm. (Minor deviation from paper; documented in JOURNAL.)
  3. MT-Bench reward: MT-Bench is held-out / report-only (10-pt LLM-judge score, kept out of any Bernoulli training reward). We never binarize it for fitness. R2's "every held-out task" check uses the 10-pt score as a ranking, not a {0,1} reward.
  4. Budget-matched single-model baselines (R1/R2): run each single model at max_tokens = 20,480 (5×) so the single-vs-TRINITY comparison is fair, matching the paper's 5× protocol.
  5. Verifier turn-1 guard: a Verifier ACCEPT terminates only if ≥1 Worker output already exists in the transcript; otherwise treat ACCEPT as REVISE. Prevents a turn-1 Verifier from accepting an empty solution (guaranteed reward-0 trajectories that bias CMA against Verifier).
  6. Reward checkers are first-class, not "OUR CHOICE niceties": they define the fitness signal. reward.py must implement: code pass@1 via a sandboxed executor with timeout; math via boxed-answer / last-number extraction + symbolic equality; MMLU/GPQA via robust letter extraction. Each gets a unit test (S5) with one known-correct + one known-wrong case.
  7. Caching × greedy × binary-reward interaction: disk cache is keyed by (model, prompt_hash, decode_params). With temp 0.0 this makes trajectories deterministic, so two CMA candidates that pick the same (agent, role) sequence on the same instance get identical rewards. This is acceptable (reduces cost) but means inter-candidate fitness variance comes only from different (agent, role) choices — monitor that candidates actually diverge.

0.4 CMA constants — which n?

The block-ε-separability that justifies sep-CMA-ES was measured on the paper's 7-agent representation. We cannot assume it transfers to a 3-model pool, so R8 (CMA > SFT > RS > REINFORCE) is a hypothesis to test, not a given. We compute λ on our actual joint n = 13,312 and use the cma library's separable defaults for the rest. If S7/early iterations show CMA not improving, the first thing to re-examine is the separability assumption on our pool (log to JOURNAL).

TRINITY Replication — Canonical Implementation Spec (SPEC.md)

Re-implementation of TRINITY: An Evolved LLM Coordinator (ICLR 2026, arXiv:2512.04695v3) for OUR setting: a 3-model OpenRouter pool coordinated by a local Qwen3-0.6B SLM, trained with sep-CMA-ES on a single H200 (GPU index 5, ~143 GB).

Source paper: /home/cybernovas/Desktop/2026/experiments/trinity/docs/paper/trinity_paper.txt. Every paper-quoted number is preserved. Where the paper is silent, a default is proposed and tagged [OUR CHOICE]. Where OUR setting deviates by design (pool size), it is tagged [REPLICATION DELTA].

1. Goal & Success Criteria

1.1 What TRINITY is

A tiny (< 20K trainable params) coordinator that, at each of up to K=5 turns, reads the full conversation transcript with a frozen-ish Qwen3-0.6B SLM, and via a ~10K-param linear head emits two decisions: which LLM to call and which role (Thinker / Worker / Verifier) it plays. The coordinator's own generated text is discarded; only the head logits matter. It is trained derivative-free with sep-CMA-ES against a binary terminal reward (task correct = 1).

1.2 OUR setup (fixed)

  • Coordinator SLM: Qwen3-0.6B, run locally on H200 GPU5. Hidden dim d_h = 1024.
  • Coordinated pool (L = 3) [REPLICATION DELTA]: OpenRouter-served qwen3.5-35b-a3b, gemini-3.1-flash-lite, deepseek-v4-flash. (Paper used L=7: GPT-5, Gemini-2.5-pro, Claude-Sonnet-4-20250514, Gemma-3-27B-It, DeepSeek-R1-Distill-Qwen-32B, Qwen3-32B reasoning, Qwen3-32B direct.)
  • Head output n_a = L + 3 = 6 logits (3 agent + 3 role). Linear head = 6 × 1024 = 6,144 params (paper: 10,240 at L=7).
  • Roles: Thinker (T), Worker (W), Verifier (V).

1.3 Success criteria — RELATIVE, not absolute

Absolute scores will differ (different pool, no GPT-5/Gemini/Claude). We replicate the paper's relative invariants (paper anchors in parentheses):

# Invariant to reproduce Paper evidence
R1 TRINITY avg > best single model avg (budget-matched 5×) 70.44 > GPT-5 65.95 (in-dist); 54.21 > Gemini 52.34 (held-out)
R2 TRINITY > every single model on every task Tables 1, 2
R3 TRINITY > best multi-agent baseline (MoA/MasRouter/RouterDC/Smoothie) §4.2, Fig.3
R4 TRINITY > random routing RLPR: 0.41 vs 0.32
R5 TRINITY ≈ Per-Question-Best on 3 of 4 in-dist tasks §4.2–4.3
R6 Lifting token cap → large LiveCodeBench jump, beats all constituents 0.61 → 0.862, beats GPT-5 0.838
R7 More max-turns → monotonic gain 0.823→0.863 (2→6 turns)
R8 sep-CMA-ES > SFT > RS > REINFORCE on all 4 tasks Table 4
R9 Removing SVF / Thinker / tri-role / penultimate-token all hurt; tri-role + token-choice matter most Table 2
R10 linear head ≥ all other head variants overall Table 3
R11 Trained coordinator > LLM-as-coordinator Table 8
R12 TRINITY far more token-efficient than MoA/Smoothie/MasRouter Table 9
R13 Mean relative-error-reduction ≈ 21.9% vs 2nd-best (ballpark, pool-dependent) §1

Definition of done for the replication: R1–R4 and R8 hold on at least 2 of our chosen in-distribution tasks; the trained coordinator runs end-to-end within the atomic-eval budget on one H200; the optimizer drives J(θ) upward over iterations.

2. System Architecture + Data-Flow Diagram

Two nested loops:

  • Inner (coordination run): one trajectory τ = one atomic Bernoulli evaluation. Up to K turns of (select agent+role → prompt LLM → post-process → append).
  • Outer (training): sep-CMA-ES samples candidate θ vectors, evaluates each by averaging m_CMA inner runs, recombines into the next parent.
                          OUTER LOOP  (sep-CMA-ES, ~60 iters)
  ┌──────────────────────────────────────────────────────────────────────────┐
  │  parent θ (mean m_t, step σ_t, diag D_t)                                   │
  │     │ sample λ candidates  y = m_t + σ_t·D_t·z,  z~N(0,I)                   │
  │     ▼                                                                      │
  │  for each candidate θ_i:  fitness = mean over m_CMA inner runs of R(τ)     │
  │     │                                                                      │
  │     ▼  fitness-weighted recombination → new m_{t+1}, σ_{t+1}, D_{t+1}      │
  └──────────────────────────────────────────────────────────────────────────┘
                                    │ θ defines (head W) + (SVF singular-value scales)
                                    ▼
   INNER LOOP  (one trajectory τ = 1 Bernoulli atomic eval)
   ┌────────────────────────────────────────────────────────────────────────┐
   │ user query Q ──► transcript C_0 = [Q]                                    │
   │                                                                          │
   │ for k = 1..K:                                                            │
   │   ┌───────────────────────────────────────────────────────────┐        │
   │   │ s = concat(C_{k-1})  (Q + all prior outputs O_1..O_{k-1})   │        │
   │   │            │                                                │        │
   │   │            ▼                                                │        │
   │   │   Qwen3-0.6B (SVF-adapted, FROZEN orthogonal factors)       │        │
   │   │   forward over [..<Head Input><EOS>]                        │        │
   │   │            │  h = hidden state @ PENULTIMATE output token   │        │
   │   │            ▼  (final layer, R^1024)                         │        │
   │   │   LINEAR HEAD  z = W·h     W∈R^{6×1024}                      │        │
   │   │            │                                                │        │
   │   │     ┌──────┴───────┐                                        │        │
   │   │  agent logits[0:3] role logits[3:6]                         │        │
   │   │     │softmax        │softmax                                │        │
   │   │     ▼               ▼                                       │        │
   │   │   A_k ∈ pool      R_k ∈ {T,W,V}                             │        │
   │   └───────────────────────────┬───────────────────────────────┘        │
   │                               ▼                                          │
   │           MESSAGE-PROCESSING MODULE: inject role-specific prompt(R_k,C)  │
   │                               │                                          │
   │                               ▼                                          │
   │             OpenRouter LLM A_k .generate(prompt, max_tokens=4096) → M_k   │
   │                               │                                          │
   │                               ▼                                          │
   │             post-process M_k → O_k ; C_k = C_{k-1} ∪ {O_k}               │
   │                               │                                          │
   │     if R_k == V and parse(O_k)==ACCEPT:  τ=k; break                      │
   │   end for  (else τ=K)                                                    │
   │                               ▼                                          │
   │             final answer = O_τ ──► reward R(τ)∈{0,1} (correctness)       │
   └────────────────────────────────────────────────────────────────────────┘

Module mapping: SLM+head = src/trinity/coordinator; pool = src/trinity/llm; role prompts/post-processing/verifier-parse = src/trinity/roles; inner loop = src/trinity/orchestration; outer loop = src/trinity/optim.

3. Coordinator: Qwen3-0.6B + Hidden-State Extraction + ~10K Head

3.1 SLM

  • Model: Qwen3-0.6B (paper §4.1). d_h = 1024 (penultimate-layer hidden size; verify against the actual checkpoint config at load — paper asserts 1024).
  • Run on H200 GPU5 in bf16/fp16.

3.2 Hidden-state extraction (exact)

  • Input construction: feed the full transcript as the SLM input. Figure 2 shows a constructed sequence <BOS> ... <Head Input> <EOS>. The head reads the penultimate output token position (the <Head Input> position) at the final hidden layer.
  • Token: penultimate output token — NOT the last/EOS token. (Ablation: using last token collapses LiveCodeBench by >10 points, 61.46→50.85.)
  • Pooling: none. Single token's vector h ∈ R^{1024} is the head's sole input. No turn index, no role one-hots, no history features.
  • The SLM's generated text is discarded; the design even permits using an earlier-than-penultimate token for speed, but all experiments use penultimate — so do we.

[OUR CHOICE] Exact mechanics: Append a fixed prompt suffix to the transcript that asks the SLM to "decide the next agent and role," run a 1-token forward (or generate up to a <think>/EOS so a penultimate position exists), and take the hidden state at the second-to-last position of the produced sequence. Standardize on: tokenize transcript + "\n<Head Input>", do one forward pass with a single appended EOS, read index -2. Document this as the canonical extraction so train and inference match.

3.3 Head (linear, default)

  • Equation 5: z = W·h, W ∈ R^{n_a × d_h}, no bias, no activation.
  • Output size n_a = L + 3.
    • Paper: L=7 → n_a=10 → W ∈ R^{10×1024} = 10,240 params.
    • [REPLICATION DELTA] OURS: L=3 → n_a = 6 → W ∈ R^{6×1024} = 6,144 params.
  • Decision mapping: split logits → first L = agent logits, last 3 = role logits. Apply softmax separately to each group (two independent categoricals); the policy is π_θ(a|s) ∝ exp(f_θ(h)_a).
  • No stop logit. Termination is implicit (Verifier selected + ACCEPT, or K reached).

3.4 The trainable parameter vector θ for CMA-ES

θ = concatenation of two sets (total < 20K):

Component Paper (L=7) OURS (L=3) [REPLICATION DELTA]
Linear head W 10,240 (10×1024) 6,144 (6×1024)
SVF singular-value scales (Qwen3-0.6B 2nd-to-last layer) 9,216 9,216 (unchanged)
Total dim n 19,456 15,360
  • SVF (Transformer² / Sun et al. 2025): SVD a selected subset of weight matrices in the second-to-last layer of Qwen3-0.6B; learn only singular-value scales, keep U, V (orthogonal factors) frozen. 9,216 scales.
  • [OUR CHOICE] Which matrices get SVF: paper does not enumerate. 9,216 = 9×1024. Default: apply SVF to the Q,K,V,O attention projections + gate/up/down MLP of the 2nd-to-last block, taking the top singular components so the learnable-scale count is 9,216; initialize all scales to 1.0 (identity). If the exact matrix set can't hit 9,216, fall to "all linear weight matrices in that block, scales=1.0" and record the actual count.

[OUR CHOICE] Optimize head + SVF jointly (single θ of dim 15,360). The paper's "<20K" total and Fig.2 both imply joint optimization; theory's n≈10000 refers loosely to the head.

3.5 Alternative heads (ablation parity, Appendix A.4)

All map h∈R^{d_h} → z∈R^{n_a}. Param counts shown at paper's d_h=1024, n_a=10:

Head Equation Params (n_a=10) Notes
linear (default) z=Wh, no bias 10,240 best overall
low-rank u=ELU(Uh, α=0.1); z=Vu·σ; r=14, σ fixed; Xavier-uniform init 20,680 gains U~U[±√(6/(d_h+r))], V~U[±√(18/(r+n_a))]
sparse z=W(h⊙α); k=max(1,⌊d_h(1−σ(ρ))⌋); Gumbel top-k τ∈[1.0,20.0]; hard top-k at inference 11,266 d_h·n_a + d_h + 2
block-diagonal-2 B=2 proportional blocks 5,120
block-diagonal-10 B=10, one block/logit, argmax output 1,024 exact 10× reduction

4. Roles + Multi-Turn Protocol + Prompt Templates

4.1 Role contracts (verbatim §3.2)

  • Thinker strategizes: returns meta-level guidance — high-level plans, decompositions, critiques of partial solutions; may propose subgoal plan; may specify the role of the next agent.
  • Worker executes: acts directly on the task; produces actionable content (derivation, code snippet, numerical result).
  • Verifier evaluates: checks if the accumulated solution is correct/complete/responsive to Q; outputs u_k ∈ {ACCEPT, REVISE} + optional diagnosis δ_k. If ACCEPT → signal termination.

4.2 Per-turn protocol

  1. Build C_{k-1} = (Q, O_1, …, O_{k-1}) — query concatenated with all prior outputs.
  2. SLM forward → h → head → softmax → pick A_k (agent) and R_k (role).
  3. Message-processing module injects role-specific prompt over C_{k-1}.
  4. Query A_k → message M_k (max 4096 tokens, minimal reasoning effort).
  5. Post-process M_k → O_k; append to transcript.
  6. If R_k = V and parse(O_k) = ACCEPT → terminate.

Termination: τ = min{ k ≤ K : R_k = V ∧ u_k = ACCEPT }, else τ = K. Final answer = O_τ. Horizon: K = B_turn = 5 (treat as identical). Decoding (global, all pool LLMs): max_tokens = 4096, "minimal reasoning effort". No temperature/top-p stated.

4.3 Inference selection rule

[OUR CHOICE] Use argmax over each softmax group at deployment/eval for reproducibility; use sampling from π_θ only inside training fitness evaluation so the optimizer sees the stochastic policy it is optimizing. (Paper: default conversion is softmax; block-diag-10 uses argmax; sampled-vs-argmax at inference is unspecified for the linear head.)

4.4 Prompt templates — NOT IN PAPER, must author [OUR CHOICE]

The paper gives only contracts + one Fig.1 example ("The next agent should act as a solver…"). No verbatim templates exist. Author these system prompts:

THINKER

You are the THINKER. Do NOT solve the task end-to-end.
Analyze the current state and produce meta-level guidance: a concise high-level
plan, a decomposition into subgoals, or a critique of the partial solution so far.
You may recommend which role should act next.
QUERY:
{Q}
TRANSCRIPT SO FAR:
{C_prev}
Return only your plan/critique.

WORKER

You are the WORKER. Make concrete progress toward the final answer.
Follow any plan in the transcript. Produce actionable content: the derivation,
the code, or the numerical/final result. Be explicit and complete.
QUERY:
{Q}
TRANSCRIPT SO FAR:
{C_prev}
Return your solution work.

VERIFIER

You are the VERIFIER. Check whether the accumulated solution is correct, complete,
and responsive to the query.
End your response with EXACTLY one line:
VERDICT: ACCEPT      (if the current answer is correct and final)
VERDICT: REVISE      (otherwise, with a one-line diagnosis above it)
QUERY:
{Q}
TRANSCRIPT SO FAR:
{C_prev}

4.5 Post-processing M_k → O_k — NOT IN PAPER [OUR CHOICE]

Paper says only "lightly post-processes / condenses / extracts." Default: pass-through with light truncation — keep M_k verbatim but cap each O_k at a fixed char/token budget (e.g. 2,000 tokens) to bound transcript growth; for Verifier turns, store (u_k, δ_k) where δ_k is the text above the VERDICT line. Do NOT add an extra summarizer LLM call (keeps atomic-eval cost predictable). Revisit only if transcripts overflow the SLM context.

4.6 Verifier ACCEPT/REVISE parsing — NOT IN PAPER [OUR CHOICE]

Parse the last VERDICT: <ACCEPT|REVISE> line (case-insensitive, regex VERDICT:\s*(ACCEPT|REVISE)). If absent, default to REVISE (fail-safe: never terminate on an unparseable verifier). This makes the ACCEPT signal deterministic and decouples it from free-text.

4.7 First-turn behavior [OUR CHOICE]

Paper does not force a first role. Let the head choose freely from turn 1 (the formal rule permits even a turn-1 Verifier-ACCEPT). No forced T→W→V ordering.

5. sep-CMA-ES (Optimizer)

5.1 Search space

θ ∈ ℝ^n, n = head + SVF (joint). OURS: n = 15,360. Paper theory: n ≈ 10000.

5.2 Fitness

  • Atomic evaluation = one full trajectory τ → Bernoulli reward R(τ) ∈ {0,1}.
  • Fitness(θ) = mean of R(τ) over m_CMA replications. Objective J(θ) = E_{τ~π_θ}[R(τ)], maximized.
  • No explicit cost/length penalty in fitness; cost is bounded structurally by max_tokens=4096 × K=5.
  • B_env counts individual Bernoulli calls. CMA cost/iteration = m_CMA · λ; T = ⌊B_env / (m_CMA·λ)⌋.

[OUR CHOICE] Replication construction: each of the m_CMA draws uses a different randomly-sampled task instance from the training set (a minibatch of 16 distinct problems per candidate), re-sampled per iteration. (Paper is ambiguous: SFT used 3 seeds, RS used 32 trials; "replication/averaging" unspecified.) This gives an unbiased low-variance estimate of J.

5.3 Hyperparameters

Param Paper value (n≈10000) OURS (n=15,360) Source / note
Algorithm sep-CMA-ES (diagonal covariance, Ros & Hansen 2008) same §3.3
Population λ ⌈4 + 3 ln n⌉ = 32 ⌈4+3 ln 15360⌉ = 34 [REPLICATION DELTA] recompute on our n
Replication m_CMA 16 16 §3.3
Parents μ symbolic only ⌊λ/2⌋ = 17 [OUR CHOICE] sep-CMA-ES default
Recombination weights w_j symbolic only default log-weights w_j ∝ ln(μ+0.5) − ln j, normalized [OUR CHOICE] Ros & Hansen default
Diag cov learning rate c_cov Θ(1/n) use library default (n+2)/3 form ≈ Θ(1/n) §A.1
c_σ, d_σ, c_1, c_μ not given library defaults [OUR CHOICE] use pycma / sep variant
Initial mean m_0 not given head W = 0 (uniform policy); SVF scales = 1.0 [OUR CHOICE] symmetric start
Initial step σ_0 not given 0.1 [OUR CHOICE] small, since W=0 start; tune if collapse
Bounds on θ none stated (RS-only band) none (free) [OUR CHOICE] clip only if divergence
Restart logic none none (single run) matches paper
Iterations T analyzed [2,60]; 60 used for budget-match target ~60 [OUR CHOICE]
Total budget B_env 1.5k–40k ≈ m_CMA·λ·T = 16·34·60 = 34,560 (within range)
Output conversion softmax (argmax for block-diag-10) softmax (train), argmax (eval) §Table 3 caption

5.4 Why this optimizer (claimed advantage)

High dim + weak parameter coupling + high per-step cost → REINFORCE per-parameter gradients are low-SNR. The objective exhibits block-ε-separability (Def. 1: scaled Hessian is near block-diagonal), so a diagonal-covariance ES is well-matched. Theory (Prop 1) gives CMA/RS gain ratio ≈ (T/ln(16T))·η² > 1 even for small T; (Prop 2) per-iteration contraction ~(κ̄_{μ,λ}/n)(1−O(ε_H)) after a Θ(n) transient.

6. Benchmarks & Eval Protocol

6.1 In-distribution (train + eval), four tasks

MATH500, MMLU, RLPR, LiveCodeBench. Train per-task, eval on the matching test set (one coordinator per benchmark — no multi-task blend [OUR CHOICE matches paper]).

  • LiveCodeBench (exact split): train on V1 (400 samples), eval on V6 newly-introduced (175 samples), Jan–Apr 2025.
  • Combined training pool ≈ 7,000 datapoints; SFT oracle used 3 seeds.
  • [OUR CHOICE] Split sizes for MATH500/MMLU/RLPR (paper says "official splits where available" but gives no numbers): use official train/test; if none, hold out a fixed 20% as test, seed=0. Document actual sizes used.

6.2 Held-out / zero-shot transfer (no retraining)

AIME2025, BigCodeBench, MT-Bench(-101), GPQA-Diamond. Same K=5 / 4096-token settings assumed.

6.3 Metrics

  • LiveCodeBench: pass@1 (execute tests).
  • MATH500 / GPQA-D / AIME: exact answer-match accuracy.
  • MMLU / BigCodeBench: accuracy / pass@1.
  • MT-Bench: ~10-point LLM-judge score ([OUR CHOICE] judge = strongest pool model, GPT-4-class rubric; keep separate from accuracy averages).
  • Reward R(τ) ∈ {0,1} per atomic eval (correctness checker per task above).
  • Relative Error Reduction: RER = (Z − S*)/(1 − S*) where Z = coordinated score, S* = best single-agent on subset.
  • Per-Question-Best upper bound = union of correct answers across the pool.

6.4 Seeds & decoding

  • [OUR CHOICE] 3 eval seeds (matches paper's only stated seed count), report mean ± std.
  • [OUR CHOICE] Decoding for pool LLMs: temperature 0.0 (greedy) for deterministic correctness scoring on math/code; top_p=1.0. "Minimal reasoning effort" → set each OpenRouter model's reasoning/thinking budget to its lowest documented setting. Document the exact per-model mapping.

6.5 Full-power LiveCodeBench

After training, lift the 4096-token cap, no retraining (R6). Paper's pool narrowed to 3 closed models; OUR pool is already 3 models, so simply remove the cap and re-eval.

7. Target / Relative Results to Reproduce

7.1 Paper anchor numbers (for sanity, not absolute targets)

In-distribution (linear head, 4096 cap, K=5): TRINITY = LiveCodeBench 61.46 (0.615) / MATH500 88.00 / MMLU 91.56 / RLPR 40.72; Avg 70.44. Held-out (Table 1): AIME 50.00 / BigCodeBench 35.80 / MT-Bench 9.60 / GPQA-D 76.82; Avg 54.21 (beats best single Gemini 52.34). Full-power LCB: 0.862 ± 0.005 (GPT-5 0.838, Gemini 0.672, Claude 0.465); turns 2→6: 0.823→0.863.

Optimizer comparison (Table 4):

LCB MATH500 MMLU RLPR
REINFORCE 0.253 0.459 0.500 0.266
RS 0.374 0.794 0.897 0.345
SFT 0.592 0.786 0.906 0.360
sep-CMA-ES 0.615 0.880 0.916 0.401

Ablations (Table 2, Avg): full 70.44; w/o SVF 67.85; w/o Thinker 68.69; w/o tri-role 67.02; last-token 64.66; GPT-5 only 65.95; Gemini only 62.99; Claude only 61.12.

Heads (Table 3): linear best overall (0.615/0.880/0.916/0.401); sparse edges MMLU only (0.917); block-diag-10 retains much at 1,024 params.

Token efficiency (Table 9, avg out-tokens): TRINITY 2,853/1,200/2,141/1,999 ≪ MoA 6,871/5,218/11,086/21,634; Smoothie similar-high; RouterDC lower but worse accuracy.

7.2 OUR targets

We do not target absolute numbers. We target the R1–R13 invariants (Section 1.3). Known paper inconsistencies to ignore: LLM-as-coordinator avg "64.14" (text) vs 53.76 (Table 8) — use 53.76; LCB rounding 61.46/0.615/0.61/61.49 — same run, target ≈ 61.5.

8. Module-by-Module Build Plan (src/trinity/...)

src/trinity/llm

  • pool.py: registry of the 3 OpenRouter models (qwen3.5-35b-a3b, gemini-3.1-flash-lite, deepseek-v4-flash) with agent IDs A0/A1/A2.
  • client.py: async OpenRouter chat client; generate(prompt, max_tokens=4096, temperature=0.0, reasoning="minimal"). Retry/backoff. [OUR CHOICE] optional disk response cache keyed by (model, prompt-hash, decode-params) — paper never mentions caching; add it to cut atomic-eval cost since training repeats instances.
  • Maps "minimal reasoning effort" to each provider's knob.

src/trinity/coordinator

  • slm.py: load Qwen3-0.6B on cuda:5; forward pass returning final-layer hidden states; extract penultimate-token vector (h∈R^1024). Implements §3.2 extraction contract.
  • svf.py: SVD the 2nd-to-last layer's selected matrices once at init; expose 9,216 learnable scales; reconstruct W' = U diag(s∘scale) Vᵀ; load scales from θ.
  • head.py: linear head W∈R^{6×1024} (and the 4 alt heads for ablations); forward(h) → (agent_logits[0:3], role_logits[3:6]); softmax/argmax converters.
  • params.py: pack/unpack θ ↔ (head W, SVF scales); report n=15,360.

src/trinity/roles

  • prompts.py: Thinker/Worker/Verifier templates (§4.4).
  • postprocess.py: M_k → O_k light-truncation (§4.5).
  • verifier.py: ACCEPT/REVISE regex parser, fail-safe REVISE (§4.6).

src/trinity/orchestration

  • run.py: trinity_run(Q, theta) → O_τ — the inner loop (Section 2 diagram), enforces K=5 and termination rule.
  • reward.py: per-task correctness checkers (pass@1 executor for code, exact-match for math, choice-match for MMLU/GPQA, judge for MT-Bench) → R(τ)∈{0,1}.
  • dataset.py: loaders + splits (incl. LCB V1/V6); minibatch sampler for m_CMA replications.

src/trinity/optim

  • cmaes.py: sep-CMA-ES wrapper (diagonal covariance), λ=34, μ=17, m_CMA=16, σ0=0.1, init mean (W=0, SVF=1.0), library defaults for the rest; iteration loop to T≈60; logs J(θ) per iteration.
  • fitness.py: evaluate candidate = mean R(τ) over 16 sampled instances (parallelized across the population × replication via async LLM calls).
  • baselines.py: REINFORCE (batch=m_CMA·λ, 60 iters), RS (U[−0.5,0.5], 32 trials/candidate, budget-matched), SFT (Adam, lr 1e-6, batch 64, frozen SLM, head-only) for R8.

Top-level

  • train.py (per-task training entrypoint), eval.py (in-dist + held-out, 3 seeds), config.yaml (consolidated hyperparameters from Section 9).

9. Consolidated Hyperparameter Table

Param Value OURS / note
Coordinator SLM Qwen3-0.6B (local, H200 GPU5)
Hidden dim d_h 1024 verify at load
Default head linear, z=Wh, no bias
Head output n_a = L+3 10 (paper) 6 (OURS)
Linear head params 10,240 (paper) 6,144 (OURS)
SVF target 2nd-to-last layer, learn singular-value scales matrices: [OUR CHOICE] Q/K/V/O+MLP, scales init 1.0
SVF params 9,216
Total trainable n 19,456 (paper) 15,360 (OURS)
Hidden-state token penultimate output token, final layer
Pooling none
Pool size L 7 (paper) 3 (OURS)
Roles T / W / V
Max turns K = B_turn 5
Max gen tokens / LLM 4096, minimal reasoning effort full-power: cap lifted
Pool decode temp / top_p not given 0.0 / 1.0 [OUR CHOICE]
Output conversion softmax (default), argmax (block-diag-10) train=softmax-sample, eval=argmax [OUR CHOICE]
Reward binary terminal R(τ)∈{0,1}
Optimizer sep-CMA-ES (diagonal cov)
Population λ 32 (⌈4+3 ln n⌉) 34 (OURS, recomputed)
Parents μ not given 17 = ⌊λ/2⌋ [OUR CHOICE]
Recombination weights not given default log-weights [OUR CHOICE]
m_CMA (replications) 16 minibatch of 16 distinct instances [OUR CHOICE]
m_RS 32 RS baseline
c_cov Θ(1/n) library default
Init mean m_0 not given W=0, SVF=1.0 [OUR CHOICE]
Init step σ_0 not given 0.1 [OUR CHOICE]
Bounds / restarts none none
Iterations T [2,60]; 60 used ~60 [OUR CHOICE]
Budget B_env 1.5k–40k ≈34,560 (16·34·60)
REINFORCE batch=m_CMA·λ=512(paper)/544(ours), 60 iters; LR/baseline/entropy not given [OUR CHOICE] defaults
RS U[−0.5,0.5], 32 trials/candidate, budget-matched
SFT Adam, lr 1e-6, batch 64, frozen SLM, head-only
Low-rank head r=14, ELU α=0.1, σ fixed, Xavier-uniform init ablation
Sparse head Gumbel τ∈[1.0,20.0], hard top-k at inference ablation
Eval seeds 3 (matches paper) [OUR CHOICE]
LCB split train V1=400, eval V6=175

10. Known Unknowns & Decisions (each with a recommended default)

# Unknown (paper silent/ambiguous) Recommended default [OUR CHOICE]
1 Role prompt templates — none in paper Templates in §4.4; iterate if a role misbehaves
2 M_k → O_k post-processing Pass-through + token-cap truncation (§4.5); no extra LLM summarizer
3 ACCEPT/REVISE parsing Require `VERDICT: ACCEPT
4 Penultimate-token mechanics Append <Head Input>+EOS, single forward, take index −2; identical at train+eval (§3.2)
5 SVF matrix subset Q/K/V/O + MLP of 2nd-to-last block, top components → 9,216 scales, init 1.0 (§3.4)
6 Joint vs head-only optimization Joint (n=15,360)
7 Inference selection: sample vs argmax Argmax at eval; sample at train fitness (§4.3)
8 Agent/role logit factorization Two independent softmaxes (separate categoricals)
9 CMA σ_0, μ, weights, m_0 σ0=0.1, μ=17, default log-weights, m_0=(W=0, SVF=1.0)
10 Per-task B_env / iterations T≈60 (B_env≈34,560) for all tasks; tune per-task if budget-bound
11 Replication minibatch construction 16 distinct random instances per candidate, re-sampled each iteration
12 Decode temp/top_p, "minimal reasoning" temp 0.0, top_p 1.0, lowest reasoning budget per OpenRouter model
13 MT-Bench judge & rubric Strongest pool model as judge, 10-point; keep separate from accuracy avgs
14 MATH500/MMLU/RLPR split sizes Official splits; else 80/20 with seed 0; record actual sizes
15 Eval seed count 3 seeds, report mean±std
16 LLM response caching Enable disk cache keyed by (model, prompt, decode-params)
17 First-turn role Free (no forced ordering); turn-1 Verifier-ACCEPT allowed
18 d_h = 1024 assumption Verify against the loaded Qwen3-0.6B config; if it differs, recompute head dims and n
19 Linear head init W = 0 (uniform start), folded into CMA m_0
20 REINFORCE LR/baseline/entropy Library defaults; only batch (=m_CMA·λ) and 60 iters are fixed

11. Smoke-test ladder = canonical build order (do NOT spend CMA budget until S1–S8 pass)

Build and verify in this order. Each rung is cheap and gates the next; the expensive 60-iteration CMA run is justified only after S8 returns a finite fitness. (From the adversarial review §E.)

# Gate What it proves Cost
S1 Load Qwen3-0.6B on cuda (GPU5); assert config.hidden_size==1024, capture num_hidden_layers==28. Run §3.2 extraction on one fixed transcript twice → identical h, h.shape==(1024,), index −2 is the intended position (not EOS). Print ‖h‖. SLM forward + deterministic penultimate extraction; calibrates σ₀. 1 GPU load
S2 Build SVF on layer 26; with all scales=1.0 assert reconstructed W' ≈ W (max abs diff < 1e-3 bf16) for every targeted matrix; print the learnable-scale count and assert it == what we pack into θ (expect 7,168 — verify, don't assume). Perturb one scale → SLM output changes. SVF round-trips to identity; resolves the #1 blocker. GPU only
S3 unpack(pack(W, svf_scales)) == (W, svf_scales); assert len(θ) == 6144 + actual_svf_count and that this equals the n fed to CMA. θ layout integrity. trivial
S4 Mock the LLM: stub Worker text + a Verifier emitting VERDICT: ACCEPT. Run trinity_run(Q, θ_random): assert K≤5, transcript grows, ACCEPT terminates (only after a Worker exists — §0.3.5), returns O_τ. Then a no-VERDICT verifier → fail-safe REVISE → runs to K. Inner loop, termination rule, verifier parser. $0
S5 Feed each reward.py checker one known-correct + one known-wrong case (math boxed answer, code pass@1 on a 2-test toy, MMLU letter). Assert {1,0}. Reward signal correctness (where silent failures hide). $0
S6 One live call to each of the 3 OpenRouter models: "minimal reasoning" param accepted, max_tokens honored, cache write/read works. Pool IDs + decode params + cache. 3 calls
S7 Run sep-CMA-ES at the real n on a synthetic deterministic fitness (−‖θ−θ*‖²) for ~10 iters; assert J increases monotonically and λ == configured (33). Optimizer loop, recombination, logging. CPU
S8 Real SLM + real LLM + real reward, m_CMA=2, λ=1, T=1. Assert finite fitness ∈ [0,1] logged and API calls ≤ 2×5. End-to-end integration proof. ~10 calls

Two extra gates before the full run: (a) for W=0 (uniform) θ, empirical agent/role selection is ~uniform over a few hundred samples (proves softmax factorization + sample/argmax paths); (b) confirm fitness repeatability under greedy+cache matches expectation (surfaces the §0.3.7 caching interaction).

Only after all gates pass: launch per-task training (train.py) on GPU 5 at λ=33, m_CMA=16, σ₀=0.1, T≈60, B_env≈31,680. Log J(θ) per iteration. Then eval.py (3 seeds) for R1–R13.

12. Implementation milestones (maps to tasks + JOURNAL discipline)

  1. M0 — Coordinator core (GPU5): slm.py (S1), svf.py (S2), head.py, params.py (S3).
  2. M1 — Orchestration (no GPU): roles/*, orchestration/run.py + reward.py (S4, S5).
  3. M2 — Pool integration: finalize llm/client.py reasoning-effort mapping + cache (S6).
  4. M3 — Optimizer: optim/cmaes.py + fitness.py (S7), then S8 integration.
  5. M4 — Baselines: optim/baselines.py (RS, SFT, REINFORCE) for R8.
  6. M5 — Train + eval at scale on GPU 5; collect R1–R13; write results to experiments/.

Every milestone appends a dated entry to docs/JOURNAL.md (findings, mistakes, decisions) per AGENTS.md §6 — including any place reality contradicted this spec.