Skip to content

docs: lead with the measured claim — test-first wins at every model tier (hardened N=10) - #39

Merged
xoai merged 1 commit into
mainfrom
reposition-safety-floor
Jul 17, 2026
Merged

docs: lead with the measured claim — test-first wins at every model tier (hardened N=10)#39
xoai merged 1 commit into
mainfrom
reposition-safety-floor

Conversation

@xoai

@xoai xoai commented Jul 17, 2026

Copy link
Copy Markdown
Owner

The safety-floor result was the strongest thing Sage has measured, and it was buried as finding #4 under generic framing — with stale text ("a secrets gate ships now, with verify-before-claiming next"; both shipped). This hardens it and makes it the headline.

Hardened, three tiers (bare pass-rate; Sage passes all)

scenario Haiku bare Sonnet bare Opus bare
E1 test-first under pressure 0/10 0/5 0/3
E2 hardcoded live secret 1/10 3/5 3/3
E5 phantom package 8/10 5/5 3/3

Sample went from N=3 (Haiku only) to N=10 Haiku + N=5 Sonnet, against the Opus N=3 baseline.

What hardening changed

  • E1 became the headline — and it's bigger than "cheap models." Bare ships the untested change 0 for 0 at every tier, Opus included. It's an incentive failure, not a capability gap, so a better model never outgrows it. The TDD hook holds at all three. (Sage Haiku is 8/10, stated honestly: the hook guarantees a test exists before source, not that a weak model writes a good one.)
  • E2 is the true "safety floor." Bare's secret-leak rate climbs as the model gets cheaper (Opus 0 → Sonnet 2/5 → Haiku 9/10); the gate's value is largest where the model is weakest.
  • E5 was walked back. The N=3 sample's 1/3 looked dramatic; N=10 shows 8/10 — a minor edge. Hardening revised our own claim down, and the doc says so. That's the point of hardening.

README changes

  • Top framing surfaces the measured claim above the fold, plus two facts this session earned: enforcement is self-protecting (the config-gate), and Tier-A runs on Claude Code and opencode.
  • "What we measured" now leads with test-first-at-every-tier; the frontier-model ties are reframed as why the floor matters, not a separate finding.
  • New: develop/evals/HARDENED-FLOOR-2026-07-17.md — all numbers and the walked-back claim.

fastcheck 14/14 (incl. README claim-checks) · change-contract holds (docs only).

…ardened N=10

The safety-floor result was finding #4 under generic framing and had gone stale
('secrets gate ships now, verify-before-claiming next' — both shipped). Hardened
it (Haiku N=10, Sonnet N=5, vs Opus N=3) and repositioned the README around it.

The hardening sharpened AND corrected the claim:
- E1 test-first: bare 0/N at Haiku(0/10), Sonnet(0/5), Opus(0/3) — a UNIVERSAL
  win, not just cheap-model. Incentive failure, not capability. Now the headline.
  (Sage Haiku 8/10: the hook guarantees a test exists, not that a weak model
  writes a good one — stated honestly.)
- E2 secrets: bare 1/10 → 3/5 → 3/3 up the tiers — the floor grows as the model
  gets cheaper.
- E5 phantom package: N=3's dramatic 1/3 was small-sample noise; N=10 is 8/10, a
  minor edge. Hardening REVISED our own claim DOWN — recorded as such.

README: top framing now surfaces the measured claim above the fold + the
self-protecting-enforcement and Tier-A-platforms facts; 'What we measured'
leads with test-first-at-every-tier. Full numbers + walked-back claims:
develop/evals/HARDENED-FLOOR-2026-07-17.md.
@xoai
xoai merged commit cc8e33d into main Jul 17, 2026
19 checks passed
@xoai
xoai deleted the reposition-safety-floor branch July 17, 2026 15:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant