The guard I shipped did not catch the thing it was written for - #9
Merged
opencdlee-dotcom merged 3 commits intoAug 24, 2026
Merged
Conversation
A 42-agent adversarial review reproduced, rather than argued, 18 defects. Three
were in the commit before this one.
The worst: StubbedTrustQualifiesOnEveryBody asserted that the conftest HELPER
returns a verdict suspicious_sig accepts. It never asserted that any test FILE
uses it — and it cannot, because the helper is per-body-correct by construction
while the defect is a body-specific LITERAL. Reverting all four
SUSPICIOUS_TRUST uses in test_outbound_subject.py back to "adhoc" left the whole
suite green on macOS: exactly the state that shipped 12 Windows failures. The
earlier demonstration that it "caught the defect in 0.06s" was real but tested
the wrong thing — it broke the helper, which the guard does consume.
The only honest detector for "a fixture hard-codes one body's spelling" is a
source scan, so NoTestHardCodesOneBodysTrustVocabulary is one. Reintroducing the
literal now names both sites with file:line:class while the old guard still
reports 2 passed.
It found nine more instances on its first run, all pre-existing, all in
test_regression.py, all ungated and stubbing macOS-only trusted verdicts on
green CI. Seven stub "apple" and convert mechanically (PUBLISHER_TRUST is
"apple" on macOS, so the macOS meaning is unchanged by construction). Two stub
"developer-id" and stay ratcheted: one is literally about vendor impersonation,
and a mechanical swap would silently change what a macOS assertion means, which
is a worse bug than the one being fixed. The ratchet may only shrink — a stale
entry fails the test.
And the same defect class, live in the product, in the file this branch is
named after: _custody_persistence inlined the macOS triple, so `publisher-stable`
was structurally unreachable on Windows and Linux and every off-mac host paid
full severity for a vendor's ordinary in-place update. publisher_sig() is now
the positive twin of suspicious_sig(), beside it, so there is one spelling of
each half of the vocabulary. Linux really does reach the rung: _classify_linux
sets authority to "dpkg:<pkg>", which is what the same-signer half compares.
Also from the review, each verified by reintroducing the defect:
- the conftest self-assert vanished under `python -O`, taking with it exactly
the "0 != N" noise it exists to prevent; it raises explicitly now
- the skip lists had no dead-entry check. The file argues carefully that a
RENAMED class fails loudly, and never mentions that an entry naming
something deleted is inert and invisible. Both halves now hold.
- suspicious_trust_for's Linux docstring claimed "broken" was representative.
_classify_linux cannot emit it; the suspicious_sig arm is dead there by
design and Linux keys on structure. Recorded rather than papered over.
- --durations=25 was credited with closing the cap-growth gap. It does not:
it reports the slowest CASES, not the step against its cap, and on the one
run that hits the cap it prints nothing at all. What it does buy is a
per-body cost profile, and it already paid for itself — the top two cases
measured 189.41s/188.25s here against 35.48s/35.74s on macOS, a uniform
~5.2x that makes the cheap local profile a usable model for the expensive
remote one.
- the harness step cap was cut 25->20 using durations measured while it ran
third behind a 24-minute suite. In its new position the first run measured
625s and 816s — outside the range it was sized against. Restored to 25 and
the real numbers recorded, because sizing a cap from another position is
the mistake this file has now made three times.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
PR #9's Linux legs failed on test_custody.py::PersistenceCustody:: test_vendor_rebuild_in_place_grades_medium, and the cause is the defect the previous commit was written to remove. The OLD custody gate compared trust against an inlined macOS triple with no platform branch — platform-BLIND, not macOS-only. A record carrying "developer-id" therefore earned `publisher-stable` on Linux and Windows too. No real record on those bodies carries that word, so the rung was still unreachable in production; but `_prec()` defaulted to it, so the fixture passed on all three CI bodies by asserting behaviour no Linux host could produce. Making the gate body-correct removed the only thing holding that test up. The scan added in the previous commit did not catch it because it matched only the dict-literal shape `"trust": "adhoc"`. `_prec(..., trust="developer-id")` is a KEYWORD DEFAULT. Widened, the scan finds 19 more sites of the same class across test_custody.py and test_regression.py; all are converted here, and `_prec`'s default is now this body's own publisher verdict so its docstring's claim to be "the shape every platform snapshot produces" is finally true. The honest failure is the verification, not the fix. The macOS run that gated the push proves nothing about this change BY CONSTRUCTION: PUBLISHER_TRUST is "apple" on macOS, so every converted site is a no-op there. The previous commit message says that outright and the push happened anyway. A platform-simulation harness that reproduces this in fifteen seconds had already been written this session, used once, and discarded as scratch. So it is a real harness now, and this commit is gated on it: the whole suite run under simulated linux and win, diffed against e0a350d's failure set. Both come back with no new failures. macOS: 1040 passed, 4 skipped. Also here: NoTestHardCodesOneBodysTrustVocabulary gains a _BY_DESIGN exemption for this module's own PublisherStableIsReachableOnEveryBody, which names each body's vocabulary in a table and flips the flags to match — the one place a body-specific word is the point rather than an accident. CustodyGrading joins the ratchet: `_target_change` defaults both sides to "developer-id" to compare TEAM identity across a target swap, and PUBLISHER_TRUST is "apple" on macOS, so a mechanical swap would change what the macOS assertion compares. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`42cfa3c` claimed "it is a real harness now, and this commit is gated on it".
The gating was true — the diffs were run and clean — but the harness itself
lived in a scratch directory and its diffstat touched three test files, none of
them it. Which is exactly the thing that commit message complains about, one
paragraph earlier, about the previous occurrence.
So here it is as `tests/simbody.py`, an opt-in `-p` plugin. It flips aegis's
platform flags BEFORE conftest binds its per-body mirrors (flipping them after
proves nothing, which is how the first attempt at this quietly measured
nothing), and makes conftest's platform gating agree.
Its docstring carries the two things that make it usable rather than
misleading:
* the absolute failure count under simulation is NOT meaningful — cases fail
here for reasons unrelated to any change (real macOS paths, a live
`codesign`, /private/tmp firmlinks). Only the DIFF against the same run on
your merge base is. The recipe is in the file.
* what it does not simulate and never will: os.sep, path parsing, case
sensitivity, file locking, subprocess behaviour. It covers a verdict, a
flag, a branch. A green run here does not replace the Windows leg; it stops
you sending the Windows leg something it will obviously reject.
`PYTHONPATH=tests` is in the usage line because pytest imports a `-p` plugin
before it puts anything on sys.path, and the version of this I first wrote
failed with "No module named 'simbody'".
CLAUDE.md points at it, and its stale test count (994) is corrected to 1040.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to
e0a350d(already green on CI run 32744015296). A 42-agentadversarial review reproduced 18 defects; three were in that commit.
The guard did not catch what it was written for.
StubbedTrustQualifiesOnEveryBodyasserted the conftest helper returns a verdict
suspicious_sigaccepts. It neverasserted any test file uses it — and cannot, because the helper is per-body-correct by
construction while the defect is a body-specific literal. Reverting the four
SUSPICIOUS_TRUSTuses back to"adhoc"left the whole suite green on macOS: exactly thestate that shipped 12 Windows failures.
NoTestHardCodesOneBodysTrustVocabularyis a source scan, the only honest detector forthis. Reintroducing the literal now names both sites with
file:line:class, while the oldguard still reports
2 passed.It found nine more instances on its first run — all pre-existing, all in
test_regression.py, all ungated and stubbing macOS-only trusted verdicts on green CI.Seven convert mechanically (
PUBLISHER_TRUSTis"apple"on macOS, so the macOS meaningis unchanged by construction). Two stay ratcheted: they stub
"developer-id"and one isliterally about vendor impersonation, so a mechanical swap would silently change what a
macOS assertion means. The ratchet may only shrink — a stale entry fails the test.
Same defect class, live in the product.
_custody_persistenceinlined the macOStriple, so
publisher-stablewas structurally unreachable on Windows and Linux and everyoff-mac host paid full severity for a vendor's ordinary in-place update.
publisher_sig()is now the positive twin of
suspicious_sig(), beside it. Linux really does reach the rung—
_classify_linuxsetsauthorityto"dpkg:<pkg>", which is what the same-signer halfcompares.
Also, each verified by reintroducing the defect:
python -O; it raises explicitly nowsuspicious_trust_for's Linux docstring claimed"broken"was representative;_classify_linuxcannot emit it, so that arm is dead there by design--durations=25was credited with closing the cap-growth gap. It does not — it reportsthe slowest cases, and on the run that hits the cap it prints nothing. What it does buy
is a per-body cost profile, and it already paid for itself: 189.41s/188.25s here against
35.48s/35.74s on macOS, a uniform ~5.2x
24-minute suite. In its new position the first run measured 625s and 816s — outside that
range. Restored to 25 with the real numbers recorded.
Local: 1040 passed, 4 skipped. Targets
feat/custody-gradingrather than pushing to itdirectly, because another agent has uncommitted work in that checkout.
🤖 Generated with Claude Code