idea: scaffold-neutral minimal-mesh baseline for eval-perf — inspired by SWE-agent's mini-swe-agent harness-neutral benchmarking

[github-actions]

Summary

Add an optional minimal-scaffold reference run to scripts/eval-perf.mjs (L4 PerfCard) — a deliberately bare mesh (fewest peers, no worker-prompt roster/skill scaffolding) run through the same routing cells alongside the full-featured mesh — so a quality/cost/latency delta between the two runs isolates mesh-orchestration effect from underlying-model capability drift. Today every PerfCard number conflates both causes.

Landscape basis

SWE-agent's maintainers now direct most effort toward mini-swe-agent, a deliberately tiny, fixed reference harness used as a control across every model comparison specifically so leaderboard deltas reflect model capability, not scaffold differences — the same underlying model can swing SWE-bench-style scores by 10-20 points purely from harness/prompting choices.

• Source: SWE-agent / mini-swe-agent
• Source: "Stop Comparing LLM Agents Without Disclosing the Harness"

Gap vs. existing perf-eval issues

scripts/eval-perf.mjs meters routing accuracy, efficiency (tokens/$, latency), and judge-scored quality per routing cell (3/6/12 peers × disjoint/confusable), always against the full production mesh scaffold (real peer prompts, real roster injection, real skills). The active perf-regression backlog (#463, #523, #690, #691, #696, and the still-open [mesh-scan] perf-regression issues) repeatedly shows metric swings without a way to attribute them to mesh-specific causes vs. model-level drift. #658 (per-agent token-budget baseline tracking) and #813 (per-cell stratification) both improve visibility into the existing numbers but neither adds a scaffold-independent control arm to separate these two causes. #797 (degenerate-metric guard) and #516 (statistical-significance gating) are also visibility/gating improvements on the same axis, not a new control-arm comparison. Confirmed by reading #658/#813/#797/#516's bodies directly (2026-07-30) — no overlap.

Proposed design

• Add a --baseline-scaffold (or config-gated) mode to eval-perf.mjs that, for a subset of trials, runs the identical routing-cell task set against a minimal reference mesh fixture: the smallest viable peer count (the 3-peer disjoint cell — already the noisiest per Stratify eval reporting by peer-overlap cell so the 3x-disjoint precision/recall regression isn't masked by aggregate gains #813/Zero-value quality_per_1k_tokens and recurring precision/recall regressions suggest an evaluator bug, not real decay #797), no roster injection (AGENT_MESH_EVAL_NO_ROSTER=1, an existing removal-only seam), and a bare-bones worker prompt (no skills, minimal AGENT.md).
• Report the minimal-scaffold run's quality/cost/latency numbers as a separate baseline row alongside the full-mesh PerfCard for the same cell — not merged into the composite score.
• When a regression is flagged (as today, via the existing MIR/perf-scan path), the daily report can then show both deltas side by side: a large gap between the full-mesh delta and the minimal-scaffold delta points at mesh-specific causes; a small gap points at model-level drift — changing where the investigation should start.
• Purely additive: no change to existing PerfCard cells, thresholds, or the production worker/peer scaffolding used elsewhere.

Invariants preserved

• No change to any served agent's prompt, tools, or write surface — the minimal-scaffold mesh is a disposable eval fixture (same pattern as existing eval/perf/ fixtures), not a production config change.
• Record-only, like the rest of L4 — no new merge-blocking gate.

Risks

Adds eval run time/cost (one more scorecard arm per cell run). Mitigated by scoping the minimal-scaffold arm to the 3-peer disjoint cell only rather than the full cell matrix.

Dedup confirmation

Checked full open-issue title/body set (2026-07-30). #658, #797, #813, #516, #527 are the nearest matches; all are visibility/gating improvements to the existing full-mesh numbers, not a scaffold-neutral control-arm comparison. No open or recently-closed issue proposes running eval-perf against a deliberately minimal reference scaffold to separate mesh-effect from model-effect.

Sources

• SWE-agent / mini-swe-agent
• "Stop Comparing LLM Agents Without Disclosing the Harness"
• agent_mesh source: scripts/eval-perf.mjs, AGENT_MESH_EVAL_NO_ROSTER (CLAUDE.md Config section)

---

Draft spec location: docs/superpowers/specs/2026-07-30-eval-perf-scaffold-neutral-baseline-design.md (Goal/Non-goals/design/invariants/risks above). Spec-push is currently blocked pending #771 (DEV_MESH_PAT not configured — reconfirmed 2026-07-30 via gh api repos/danabaxia/agent_mesh --jq '.permissions', still push: false, active token still the integration/15368 app install token). This issue carries the full draft in the body in the meantime, matching current practice for #818/#824/#831/#849/etc.