[RFC] Measuring scaffolding-dependence: a harnessability taxonomy + ablation for spec search & self-improving operators

[gilbert-barajas]

(Discussion issue — proposing an evaluation methodology + framework that feeds the Learning roadmap. Research-flavored; offered as a collaboration, n=1 prelim flagged honestly.)

Motivation

Three roadmap/feature lines all optimize a harness from traces but lack a shared measurement of how much harness a given model actually needs:

• LLM-guided spec search (user-guide) — frontier model analyzes traces, proposes config/spec improvements.
• Self-improving operators (Research-Stage) — operators tune their own prompts + routing from trace feedback.
• Learning (architecture/learning) + the evals framework, which already treats energy / FLOPs / latency / dollar cost as first-class.

To optimize a spec and know it helped, you need to answer, per capability: how much does this local model supply on its own vs. need scaffolded — and which scaffolding actually recovers the capability? We propose adding scaffolding-dependence as a first-class eval axis, with a taxonomy and a controlled instrument.

Proposal

(a) The harnessability ladder — a taxonomy of the properties a task needs, ranked by how deterministically each can be supplied or checked:

1. oracle-able (a deterministic check exists — compile/test; scope gates; domain-semantic correctness via metamorphic property tests),
2. retrievable (surface the right context — signature/spec retrieval),
3. substrate-encodable (move the knowledge into what the model reads — e.g. metric semantics in docstrings),
4. critique-able (no clean oracle, adversarial review catches most),
5. residue (open-ended judgment — the genuine frontier premium).

This tells spec-search where a property can be moved (promote runtime judgment → a compile-time oracle).

(b) The spec-tightness × model-role ablation — hold model + task fixed; vary spec detail (full vs intent-only) and model role (coder vs reasoner). The delta quantifies, per property, how much the model supplies itself vs needs scaffolded — a reproducible signal the spec-search loop can optimize against and self-improving operators can use as a reward.

(c) Preliminary result (n=1, honest). On an intent-only spec, a 30B coder produced grounded, scoped, building, tested code with a semantically wrong decision rule (ignored a metric whose "good" direction is inverted). A 27B reasoner, identical brief, produced the correct multi-metric rule and the exact adversarial test the coder missed. Findings: (i) semantic design is role-dependent — route it to the reasoner; (ii) the specific failure is oracle-able via a metamorphic property test → promotable from rung-4 to rung-1.

Mapping to your roadmap

your line	what this adds
LLM-guided spec search	a per-property signal for what to scaffold + where it can be promoted
self-improving operators	a reward / eval signal for prompt + routing tuning
evals framework	a new first-class axis (scaffolding-dependence) + scorers
leaderboard	per-property harnessability scores per model (alongside energy/cost)

What we bring

• The taxonomy + the ablation methodology + initial results.
• MOTU — an open compound-AI benchmark (a controlled task environment for "how well does a model run a real workload") we can adapt as a host for the ablation.
• Ties to compound-AI-systems + the Minions line (already on your hybrid-inference roadmap).
• Gilbert Barajas (@gilbert-barajas), active OpenJarvis contributor (6 PRs + multiple dogfood-found issues across the streaming / persona / apple_fm clusters); could anchor a joint writeup.

Open questions (for co-design)

1. How should the ablation slot into your evals/scorer interfaces — a new scorer type, or a harness around existing ones?
2. Land the taxonomy in architecture/learning docs first, or as an eval contribution first?
3. Which of your 40+ datasets are the right substrate to scale this past n=1?

Honest notes

• n=1 is a pilot. The framework (ladder + ablation) is the contribution; the numbers need scaling on your dataset suite + infra to be a claim. Offered as research collaboration, not a finished result.

[gilbert-barajas]

A second data point for (c) — on a live feature request, and it produced mergeable code

We ran the ablation lens from this RFC against a real issue (#380, per-invocation persona scope) instead of a synthetic task, to see whether the role-split finding in (c) replicates and whether the loop yields a patch a maintainer would actually take. Same instrument: a local 27B reasoner (planner — Qwen3.6-27B, dense) and an 80B-A3B coder (executor — Qwen3-Coder-Next, ~3B active, 8-bit) behind a managed-loop harness with a verification gate; spec-tightness × model-role held in view. (The coder is the current squad executor — upgraded from the 30B-A3B used in (c) after an Aider-Polyglot Python A/B, 29%→76% pass@2; the swap held active params roughly constant (~3B) and tripled the total pool (30B→80B). So this is a fresh data point with the current squad, not a re-run of (c)'s exact setup — which makes the role-split holding across the coder upgrade more interesting, not less.)

Cell tested: reasoner-on-intent → coder-on-full-spec. The reasoner got the issue + the relevant source (intent-level input) and authored a full implementation spec; the coder implemented that spec.

Result — the same shape as (c), plus a new failure mode

• Reasoner (27B) independently converged on the design reached in the human discussion — and, unprompted, added a path-traversal guard on persona names that neither the issue nor our own draft had flagged. (c) replicates on a new task and extends from semantic correctness to security judgment: route anticipatory/judgment work to the reasoner.

• Coder (Qwen3-Coder-Next, 80B-A3B) implemented the full design correctly — config field, resolver, the security guard, the CLI option — with exactly one miss: it did not thread a new parameter across a function boundary the spec called out, leaving a dangling reference (NameError). This is a distinct failure mode from (c): not semantic (the 30B coder's inverted decision rule in (c)) but structural — cross-function plumbing. Notable that the stronger coder still produced a structural-binding miss, which is the rung-1 catch below.

• The miss is oracle-able (rung-1). A dangling name fails at import/compile; it was caught deterministically by the verification phase running the module + tests. So this coder-failure class promotes cleanly to rung-1: a Python import/typecheck gate catches it with zero model judgment. In our run that catch was manual — our automated efficacy gate currently covers Swift, not Python — which is itself the actionable finding (below).

Ladder mapping for #380

property	rung	who supplied / what caught
resolution rule + precedence (semantic design)	4–5 critique / residue	reasoner — converged with the human discussion
path-traversal guard (security)	4 critique-able	reasoner, unprompted
core implementation	—	coder
cross-function binding (the bug)	1 oracle-able	import/typecheck — promotable to an automatic gate
line-length / format	1 oracle-able	ruff (already a gate)

The clean signal is the role split: reasoner → judgment + security; coder → implementation; harness → structural oracle. Same shape as (c), now with the coder's residual error living on rung-1 rather than rung-4 — which is the optimistic direction for spec-search (a real failure mode that is promotable to a deterministic check).

It closed the loop — two fixes for review

The methodology produced mergeable code, not only a measurement. Two diffs, smallest-first:

Fix A — the squad's output (per-invocation scope). --persona NAME / --persona none + a [memory_files].persona_name config field; resolution centralized in SystemPromptBuilder._resolve_persona; default (empty) preserves today's global-persona behavior exactly; includes the reasoner's guard. (4 files, +90/−5, 8 tests.)

# SystemPromptBuilder._resolve_persona — the reasoner's design + guard
if not mf.persona_name:                     # "" -> global default, unchanged
    return mf
if mf.persona_name == "none":               # explicit opt-out -> no persona
    return MemoryFilesConfig(soul_path="", memory_path="", user_path="",
                             nudge_interval=mf.nudge_interval)
name = mf.persona_name
if ".." in name or "/" in name or "\\" in name or name.startswith("/"):
    raise ValueError(f"Invalid persona name {name!r}: must be a simple "
                     "identifier (no path separators or '..').")
base = Path.home() / ".openjarvis" / "personas" / name
return MemoryFilesConfig(soul_path=str(base / "SOUL.md"), ...)

Fix B — the complete design from the discussion (the multi-engine case). Adds [engine.personas] (engine key → persona name), applied when that engine answers, precedence --persona > engine persona > config > global. Selection is split into a small prompt.persona module so every surface (CLI ask, the HTTP route) picks the name identically; path-resolution stays in the one resolver. On main the routing unit is the engine key; the same map keys on the instance name once named multi-instance engines exist. (+4 files, +117/−9, 9 tests.)

# prompt.persona — name SELECTION (paths still resolved by the builder)
def effective_persona_name(cli_persona, config, engine_key):
    if cli_persona is not None:        # explicit --persona / --persona none wins
        return cli_persona
    return engine_persona(config, engine_key) or None   # else the engine's, else honor config

Honest caveats

• n=1 on this task (a different, real task from (c) — not yet the full ablation, which holds model+task fixed and sweeps both axes). This is one cell: reasoner-on-intent feeding coder-on-full-spec. The spec the coder saw was full, so it isolates "coder executes a reasoner's complete spec," not coder-on-intent.
• The role split is the transferable signal; the convergence-with-humans and the unprompted security guard are the notable observations.

Offer

Fix A is up as #493 — the minimal mergeable change (per-invocation --persona, default-preserving, 8 tests, clean against main). Fix B (the per-engine [engine.personas] extension — the multi-engine case) is ready as a follow-up on request. Happy also to contribute the Python import/typecheck gate as a concrete rung-1 instrument for the evals framework — the smallest piece of scaffolding that converts this whole coder-failure class from runtime residue to a compile-time oracle.