fix(agent): make "verified" unearnable without verification - #30
Merged
Conversation
…autonomous-qa-enhancements] Four gaps let the product promise be claimed without being earned: - finished was inferred from a field both adapters always set, so a give-up scored as success. Completion is now only an explicit finish tool call; DriveResult carries stopReason (finished/gave_up/ step_budget) and the model's parting text, and give-ups reach the failure-learning store. - an assertion-free session compiled to a trivially-green spec and passed the gate. runQa now reports it unverified without compiling, and emitSpec refuses such sessions outright. - emitted role/text/label locators matched by substring; they are now exact on both record and replay (recorder and emitter change together), and the locator guidance says to copy names verbatim. - a numeric graph schema_version bypassed the compatibility guard as 'omitted' (now stringified and checked), an empty Playwright report mapped to a misleading 'failed' (now a diagnostic naming testDir), and configured retries inverted run semantics (now each test's final attempt is its outcome). Signed-off-by: Tom Ballard <tom@armytage.co>
…ap:autonomous-qa-enhancements] Adds tests/drive-loop.test.ts — the loop's control logic (give-up vs finish vs step budget, tool withholding, dispatch refusal feedback) unit-tested with a fake page and scripted models, so the semantics the product promise rests on run in default CI, not only behind PROOFKEEPER_E2E. Pins the emitter's assertion-free refusal and exact locators, the numeric schema_version guard, final-attempt retry reduction, and the no-tests-matched diagnostic; drive doubles now record an assertion and a stopReason. Signed-off-by: Tom Ballard <tom@armytage.co>
…oadmap:autonomous-qa-enhancements] Implements lore-proofkeeper/requirements/req-verified-semantics.md. Adds the requirement and design artifacts, the roadmap initiative, and the changelog entry. Signed-off-by: Tom Ballard <tom@armytage.co>
This was referenced Jul 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Batch 2 of the technical-review fixes. Stacked on #29 (both touch the drive loop) — merge #29 first; GitHub will retarget this to
mainautomatically.Four gaps let the product promise — the word "verified" — be claimed without being earned. Each is closed at its narrowest choke point.
What
finished = response.done !== undefinedwas vacuously true (both adapters always setdonewhen there are no tool calls), so a model that gave up scored as success and never reached failure-learning. Completion is now only an explicitfinishtool call;DriveResultcarriesstopReason(finished/gave_up/step_budget) plus the model's parting text, and give-ups are recorded to the learning store with distinct reasons.runQanow reports it unverified without compiling (QaResult.loopis optional;unverifiedReasonsays why), andemitSpecrefuses assertion-free sessions the way it already refused empty ones.getByRole/getByText/getByLabelmatched by substring, case-insensitive — a later DOM addition could silently re-target a committed test. Now{ exact: true }in the emitter and the Recorder (record/replay agreement is the invariant, changed together), with locator guidance telling the model to copy names verbatim.schema_versioncoerced to""and bypassed the compatibility guard as "omitted" — now stringified and checked (2refused,1accepted). An empty Playwright report mapped to a misleading "failed"/"unstable" quarantine — now aReportParseErrornaming the likely cause (spec outside the config'stestDir). A target project's configured retries inverted run semantics (flake-then-pass read as failed) — now each test's final attempt is its outcome.New default-CI coverage
tests/drive-loop.test.tsunit-tests the loop's control logic with a fake page + scripted models — give-up vs finish vs step budget, tool withholding, refusal feedback. This logic previously ran only behindPROOFKEEPER_E2E, which is exactly where the finished-tautology bug lived.Corpus (dogfood)
req-verified-semantics+design-verified-semantics, roadmap initiative,## Verified By→ the four pinning test files. Gates green:rac validate(31/31), relationships 45/45 (0 issues), review clean, self-coverage 15/15.Verification
PROOFKEEPER_E2E=1): all 7 integration suites pass — record→emit→fidelity stays green with exact locators, extension drive included.npm run typecheck/buildgreen.