Repository navigation
QA re-checks only what changed since its last pass - #565
Merged
Merged
Conversation
QA reads a prior receipt from an earlier head on the same branch, diffs the trees, re-runs failed, blocked, touched, changed-criterion and one smoke scenario, and carries the rest with their original head and evidence. qa receipt records per-scenario outcomes and copies carried rows from the prior receipt, refusing any that did not pass there. Task: fn-301-qa-re-checks-only-what-changed-since.1 Claude-Session: https://claude.ai/code/session_01JByA3xCfmV1eQumDVW2R3J
Any added, removed or reworded criterion since the prior pass now runs every scenario, as the spec's full-run rule requires; the per-criterion re-run and carry exceptions it made redundant are gone. Task: fn-301-qa-re-checks-only-what-changed-since.1 Claude-Session: https://claude.ai/code/session_01JByA3xCfmV1eQumDVW2R3J
The re-check pointer sat at the end of Phase 2, after scenarios were built. In 2 of 8 measured re-check runs the agent drove every scenario before it reached it, so nothing carried. Phase 1 now prints the prior receipt. Claude-Session: https://claude.ai/code/session_01JByA3xCfmV1eQumDVW2R3J
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 35a32fb. Configure here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Why
After a fix, QA no longer re-runs every scenario. In discovery, before it plans scenarios, QA prints any earlier QA receipt for the spec. When the spec already has a QA receipt from an earlier head on the same branch, QA diffs the trees between that head and HEAD. It re-drives scenarios that failed or were blocked, scenarios on touched surfaces (including their shared dependencies), and one smoke scenario, and carries every other pass with the head and evidence it was captured at. flowctl stamps live rows with the current head and copies carried rows from the prior receipt, so a carried result can never claim the new head.
What changes for a user or operator
Second and later QA passes on a spec can skip scenarios a fix could not have changed. The receipt now lists each scenario as live at this head or carried from an earlier one, and coverage gains a `carried (@sha8)` state. Without a usable prior receipt, or after a broad shared-code diff, a changed acceptance contract, a failing smoke run or unclear dependencies, QA runs in full as before.
Scope
13 files changed; +437/-27 lines; 4 generated, 3 mechanical.
1. Why re-check
A fix after a QA pass used to trigger a full re-run of every scenario. Check that the carry rules match the spec's R1-R4 and that unproven R5 sits in Open items.
2. What re-runs and what carries
Check that QA looks for the prior receipt in Phase 1, before planning scenarios, and that recheck.md re-drives everything a fix could reach and falls back to a full run whenever in doubt.
plugins/flow-next/skills/flow-next-qa/references/recheck.md: New gated reference: prior-receipt match, tree diff, full-run triggers, what re-runs, smoke first, how to record carried rows. [R1, R2, R4]plugins/flow-next/skills/flow-next-qa/workflow.md: Adds §1.4 (print the prior receipt, then read recheck.md before deriving scenarios), the carried coverage state, SHIP counting carried R-IDs, and the scenarios payload. [R1, R3]3. Receipt guards in flowctl
Check that live rows always get the current head, carried rows are copied (never trusted from the payload), and every refusal case errors.
plugins/flow-next/scripts/flowctl.py: qa receipt accepts scenarios; _qa_receipt_scenarios stamps live rows, copies carried ones; _qa_prior_scenarios refuses missing, other-spec or other-branch receipts. [R3]plugins/flow-next/tests/test_artifact_writers.py: Two behaviour tests: carried rows keep their original head and evidence; failed, absent, same-head and cross-branch carries are refused. [R3]plugins/flow-next/docs/flowctl.md: Documents the scenarios field, carry copying and refusals. [R3]4. Changelog and measured result
Check that the entry states the measured result and its limit, not a general speed-up.
CHANGELOG.md: Unreleased entry describing the re-check and the study numbers, including that the saving did not clear the noise bar. [R5]5. Mirrors and flow state
Generated Codex copies and manifest, plus the spec and task records; skim.
3 mechanical files; 4 generated files
Coverage: R1 → groups 1, 2; R2 → groups 1, 2; R3 → groups 1, 2, 3; R4 → groups 1, 2; R5 → groups 1, 4
Blast radius
Read recheck.md first: it decides what carries. Then the qa receipt writer in flowctl.py (`_qa_receipt_scenarios`, `_qa_prior_scenarios`) and its two tests. The reference is gated, so QA without a prior receipt reads nothing new. Codex mirror copies and MANIFEST.json are regenerated. Unproven: the time saving (see Open items); QA did not run on this PR (pipeline.qa is off).
Verification
Tradeoffs
Decisions taken unattended: (1) R1 against R2: a changed acceptance contract now always runs QA in full, even when the changed criterion maps cleanly to one scenario (review finding, fixed in 7e4ec8d). (2) The R5 study's pre-registered 120 s gap assumed 15-30 min QA runs; a full pass took about 2.5 min, so a secondary 20%-of-baseline gap was logged as an amendment before any arm result. The primary rule stayed the verdict. (3) After pass 1 failed on a fix that touched shared code, a second pass was pre-registered on a one-file fix, the shape where carrying should help most. Pass 1's result stands. (4) The re-reviewer marked R5 met from the changelog numbers; I record it as not met, because the pre-registered rule says so. (5) On the maintainer's call after pass 2, the prior-receipt check moved from the end of Phase 2 to Phase 1 (1237d52) and a third pass was pre-registered and run; a first pass-3 run that loaded no plugin was discarded as a harness error. (6) QA skipped: pipeline.qa is off. Completion review skipped: one task covering R1-R5 that reached SHIP. Rejected alternative: carrying on a changed-file match alone, which misses shared dependencies.
Open items
R5 is NOT MET by its pre-registered rule in any of three passes, so landing is a product call. Private study, Sonnet, a synthetic storefront where a full QA pass takes about 2.5 min, 26 valid draws. Recall held in every draw: the re-check never missed the planted regression. Time: shared-file fix 133 vs 147 s (N=3), not confirmed; one-file fix 113 vs 150 s (N=5), inconclusive because one draw skipped the re-check; one-file fix with the Phase 1 check 125 vs 145 s (N=5), gap 20 s below the 29 s bar. The Phase 1 check fixed the skipped step (5/5 draws used the prior receipt); two draws still chose a full run after finding a new P1, which the reference allows. The scenarios that carry here are the cheapest to drive, so the saving is small on this app; larger apps are unmeasured. Options: land with the measured result as stated, or park.
https://claude.ai/code/session_01JByA3xCfmV1eQumDVW2R3J