Skip to content

QA re-checks only what changed since its last pass - #565

Merged
gmickel merged 10 commits into
mainfrom
fn-301-qa-re-checks-only-what-changed-since
Oct 17, 2026
Merged

gmickel merged 10 commits into
mainfrom
fn-301-qa-re-checks-only-what-changed-since

Conversation

@gmickel

@gmickel gmickel commented Oct 16, 2026 •

Copy link
Copy Markdown
Owner

Why

After a fix, QA no longer re-runs every scenario. In discovery, before it plans scenarios, QA prints any earlier QA receipt for the spec. When the spec already has a QA receipt from an earlier head on the same branch, QA diffs the trees between that head and HEAD. It re-drives scenarios that failed or were blocked, scenarios on touched surfaces (including their shared dependencies), and one smoke scenario, and carries every other pass with the head and evidence it was captured at. flowctl stamps live rows with the current head and copies carried rows from the prior receipt, so a carried result can never claim the new head.

What changes for a user or operator

Second and later QA passes on a spec can skip scenarios a fix could not have changed. The receipt now lists each scenario as live at this head or carried from an earlier one, and coverage gains a `carried (@​sha8)` state. Without a usable prior receipt, or after a broad shared-code diff, a changed acceptance contract, a failing smoke run or unclear dependencies, QA runs in full as before.

Scope

13 files changed; +437/-27 lines; 4 generated, 3 mechanical.

1. Why re-check

A fix after a QA pass used to trigger a full re-run of every scenario. Check that the carry rules match the spec's R1-R4 and that unproven R5 sits in Open items.

2. What re-runs and what carries

Check that QA looks for the prior receipt in Phase 1, before planning scenarios, and that recheck.md re-drives everything a fix could reach and falls back to a full run whenever in doubt.

3. Receipt guards in flowctl

Check that live rows always get the current head, carried rows are copied (never trusted from the payload), and every refusal case errors.

4. Changelog and measured result

Check that the entry states the measured result and its limit, not a general speed-up.

  • CHANGELOG.md : Unreleased entry describing the re-check and the study numbers, including that the saving did not clear the noise bar. [R5]

5. Mirrors and flow state

Generated Codex copies and manifest, plus the spec and task records; skim.

3 mechanical files; 4 generated files

Coverage: R1 → groups 1, 2; R2 → groups 1, 2; R3 → groups 1, 2, 3; R4 → groups 1, 2; R5 → groups 1, 4

Blast radius

Read recheck.md first: it decides what carries. Then the qa receipt writer in flowctl.py (`_qa_receipt_scenarios`, `_qa_prior_scenarios`) and its two tests. The reference is gated, so QA without a prior receipt reads nothing new. Codex mirror copies and MANIFEST.json are regenerated. Unproven: the time saving (see Open items); QA did not run on this PR (pipeline.qa is off).

Verification

  • Implementation review: Codex: NEEDS_WORK (2 kept findings), both fixed, SHIP; Phase 1 change reviewed separately, SHIP
  • Full suite: 4054 tests, 0 failures at adf78d6; focused QA and prose tests green after 1237d52
  • Lint: ruff 0.16.0 clean
  • Codex mirror: sync-codex --check clean
  • R5 regression recall: Planted regression caught in 26/26 valid draws, both arms, three passes
  • fail: R5 time, shared-file fix: 133 s vs 147 s mean (N=3); not confirmed
  • fail: R5 time, one-file fix: 113 s vs 150 s mean (N=5); spread 38 s > gap 36.8 s: INCONCLUSIVE
  • fail: R5 time, Phase 1 check: 125 s vs 145 s mean (N=5); gap 20.2 s < 29 s bar: NOT CONFIRMED
  • unverified: QA: not run: pipeline.qa is off

Tradeoffs

Decisions taken unattended: (1) R1 against R2: a changed acceptance contract now always runs QA in full, even when the changed criterion maps cleanly to one scenario (review finding, fixed in 7e4ec8d). (2) The R5 study's pre-registered 120 s gap assumed 15-30 min QA runs; a full pass took about 2.5 min, so a secondary 20%-of-baseline gap was logged as an amendment before any arm result. The primary rule stayed the verdict. (3) After pass 1 failed on a fix that touched shared code, a second pass was pre-registered on a one-file fix, the shape where carrying should help most. Pass 1's result stands. (4) The re-reviewer marked R5 met from the changelog numbers; I record it as not met, because the pre-registered rule says so. (5) On the maintainer's call after pass 2, the prior-receipt check moved from the end of Phase 2 to Phase 1 (1237d52) and a third pass was pre-registered and run; a first pass-3 run that loaded no plugin was discarded as a harness error. (6) QA skipped: pipeline.qa is off. Completion review skipped: one task covering R1-R5 that reached SHIP. Rejected alternative: carrying on a changed-file match alone, which misses shared dependencies.

Open items

R5 is NOT MET by its pre-registered rule in any of three passes, so landing is a product call. Private study, Sonnet, a synthetic storefront where a full QA pass takes about 2.5 min, 26 valid draws. Recall held in every draw: the re-check never missed the planted regression. Time: shared-file fix 133 vs 147 s (N=3), not confirmed; one-file fix 113 vs 150 s (N=5), inconclusive because one draw skipped the re-check; one-file fix with the Phase 1 check 125 vs 145 s (N=5), gap 20 s below the 29 s bar. The Phase 1 check fixed the skipped step (5/5 draws used the prior receipt); two draws still chose a full run after finding a new P1, which the reference allows. The scenarios that carry here are the cheapest to drive, so the saving is small on this app; larger apps are unmeasured. Options: land with the measured result as stated, or park.

https://claude.ai/code/session_01JByA3xCfmV1eQumDVW2R3J

QA reads a prior receipt from an earlier head on the same branch, diffs
the trees, re-runs failed, blocked, touched, changed-criterion and one
smoke scenario, and carries the rest with their original head and
evidence. qa receipt records per-scenario outcomes and copies carried
rows from the prior receipt, refusing any that did not pass there.

Task: fn-301-qa-re-checks-only-what-changed-since.1

Claude-Session: https://claude.ai/code/session_01JByA3xCfmV1eQumDVW2R3J
Any added, removed or reworded criterion since the prior pass now runs
every scenario, as the spec's full-run rule requires; the per-criterion
re-run and carry exceptions it made redundant are gone.

Task: fn-301-qa-re-checks-only-what-changed-since.1

Claude-Session: https://claude.ai/code/session_01JByA3xCfmV1eQumDVW2R3J
The re-check pointer sat at the end of Phase 2, after scenarios were built.
In 2 of 8 measured re-check runs the agent drove every scenario before it
reached it, so nothing carried. Phase 1 now prints the prior receipt.

Claude-Session: https://claude.ai/code/session_01JByA3xCfmV1eQumDVW2R3J

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 35a32fb. Configure here.

Comment thread plugins/flow-next/skills/flow-next-qa/workflow.md Outdated
* origin/main:
  Worker reasoning effort follows the dispatch tier (#566)
  Measured numbers you can trust: limiter, errors, work done (#564)

# Conflicts:
#	CHANGELOG.md
#	plugins/flow-next/scripts/flowctl_tracker/MANIFEST.json
@gmickel
gmickel marked this pull request as ready for review October 17, 2026 08:36
@gmickel
gmickel merged commit e322da9 into main Oct 17, 2026
13 checks passed
@gmickel
gmickel deleted the fn-301-qa-re-checks-only-what-changed-since branch October 17, 2026 08:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant