Repository navigation
[Quality] Correctness prevention: SQL semantics, lifecycle testing and bounded Verus proofs #1225
Description
Recent correctness reports motivate a focused follow-up to the existing quality work: expand the semantic cases tests can reach, strengthen what a passing test establishes, exercise recovery histories, and add small production-used Verus proofs.
This issue tracks proposed quality improvements from the October 22 research review. It does not claim new runtime reproductions, reopen the entire historical Verus roadmap, or make every future proof an immediate release gate.
Work items and recommended order
First: strengthen executable tests
- [Quality] Expand DVM generators and regression coverage for PostgreSQL semantic boundaries #1217 — Expand SQL-semantic generators and permanent regressions: composite NULLs, three-valued transitions, exceptional values, evaluation protection, histories and consumer order.
- [Quality] Strengthen result oracles, negative controls and changed-data execution witnesses #1218 — Strengthen result observations, expected negative-control failures, changed-data route witnesses and local semantic mutants.
- [Quality] Add bounded lifecycle and deterministic concurrency tests for recovery #1219 — Add bounded lifecycle histories and witnessed concurrency/recovery schedules using existing Testcontainers barriers.
Next: deliver proofs in bounded production slices
- [Quality][Verus] Verify production predicate visibility and aggregate lifecycle decisions #1220 — First proof pilot: predicate visibility and aggregate lifecycle decisions.
- [Quality][Verus] Verify CDC normalization and observation-safe change elision #1221 — CDC normalization and observation-safe change elision, including protected consumer windows.
- [Quality][Verus] Verify receipt batching, cleanup eligibility and recovery transitions #1222 — Receipt batch completeness, cleanup eligibility and recovery transitions.
- [Quality][Verus] Verify a restricted join plan checker with snapshot and expression safety #1223 — Restricted join-plan checking with snapshot provenance and expression safety.
- [Quality][Verus] Verify bounded row-identity framing and encoding #1224 — Lower-priority row-identity framing and bounded encoding.
This is a prioritization recommendation, not a serial execution requirement. The SQL tests and protocol tests provide runtime evidence for the corresponding proof slices. The source-to-SQL and database boundaries must remain visible even after a proof passes.
Relationship to existing work
- bug(dvm): COUNT(composite) drops non-NULL records containing NULL fields #1203, bug(dvm): aggregate FILTER evaluates excluded arguments and raises errors #1204, bug(dvm): HAVING UNKNOWN transition leaves stale group after differential refresh #1205, bug(dvm): deleting numeric Infinity corrupts differential SUM with NaN #1209, bug(cdc): numeric scale changes are lost by equality-based changed_cols mask #1210 and bug(dvm): inner JOIN delta evaluates invalid cross-snapshot predicates #1211 retain ownership of the six reports in BUG: Several correctness issues in differential refresh #1201. Reuse their regressions; these quality issues target broader prevention.
- bug: WAL UPDATE-to-NULL retains the old decoded column value #1172–Scheduler pause can report drained while an inline refresh is still active #1187 provide additional lifecycle/capture leads with their existing evidence and confirmation status. A source-supported report is not automatically a reproduced defect.
- [Quality] Trustworthy results, recovery, and releases #1090 narrowed the previous quality program; Epic: [Verus] Deliver production-connected verified correctness in safe slices #1056 and its historical proof issues were deferred, not all implemented. The new issues revisit explicitly bounded opportunities prompted by the recent reports.
- ci: rebalance PR correctness coverage across component, E2E, and nightly tiers #1188 owns CI allocation work. Current CI already runs execution-backed DVM component tests. Extend those tests for cheap semantic combinations, retain installed-extension coverage for parser/capture/scheduler behavior, and use scheduled jobs for larger products.
- Preserve Make public-result and schema comparisons trustworthy #1091–Exercise critical unsafe boundaries under assertions and sanitizers #1098's existing oracles, runtime checks, LSN proof and candidate-binding mechanisms rather than creating parallel evidence infrastructure.
Acceptance criteria (proposed)
- R1: Each linked issue records its scoped outcome, actual verification evidence and remaining limits. Correct baselines pass; representative semantic defects are rejected for the intended reason.
- R2: Existing regression protection is retained and new checks run in the documented CI tier. Empty selection, skipped paths, fallback masking and infrastructure errors cannot masquerade as semantic coverage.
- R3: Every claimed proof uses the production executable implementation, enforced caller assumptions and a reviewed specification. SQL lowering, PostgreSQL semantics, transactions and FFI remain separately qualified where unproved.
- R4: Progress and release wording distinguish implemented tests, model proofs, executable proofs and runtime-qualified behavior. Partial delivery can ship without implying whole-extension verification.
Verification — PROPOSED — NOT RUN
Use each work item's focused verification plan and existing CI/release records. Record candidate revision, material configuration, actual runtime path and semantic-control outcomes. Follow repository formatting/linting and SQL-facing testing requirements for implementation changes.
Evidence and limits
Research baseline: 00d21749bc20af8b19940b857fae461e0ccfe471. The linked issues include inspected source locations and primary PostgreSQL/Verus references.
The preparatory review was read-only and sampled; no project code, tests, builds or proofs were executed. Acceptance criteria are proposals, and no separate saved Intentline contract was created.
Current delivery status (2026-11-01)
- [Quality][Verus] Verify CDC normalization and observation-safe change elision #1221 remains open. PR #1310 merged at
8c9c86b2b7758682127fb0bf07b2f752c8d3ca75from exact head2b999b2225642f57ba12d95e444baff50081fada. Exact-head CI run 41315604683 passed after the sensitive E2E shard rerun; S1 qualification run 41315604568 passed. The bounded slice does not complete [Quality][Verus] Verify CDC normalization and observation-safe change elision #1221: PostgreSQL reduction/capture semantics, valid multi-row history enforcement and all protected consumer-window equivalence remain open. - regression(dvm): nested inner USING joins break Delta V1 conformance after #1197 #1198 remains open. PR #1199 is at exact head
9ba4d32c574018a050a3b54f923219d4b7a5fa12, has no review yet, and still requests the sole-owner source review from BaardBouvet. - [Quality][Verus] Verify a restricted join plan checker with snapshot and expression safety #1223 and the other completed linked quality issues are closed. Keep this umbrella open until [Quality][Verus] Verify CDC normalization and observation-safe change elision #1221 and its remaining acceptance are complete.
Activity
grove commentedon Oct 29, 2026
Progress since the merged #1232 integration candidate: #1220 is now closed against its narrow acceptance using the pinned Verus baseline, five rejected executable mutants, packaged PostgreSQL 18 lifecycle cases, and candidate-binding evidence from CI run 40853118868. #1217 and #1222 are also closed. #1218, #1219, #1221, #1223, and #1224 remain open with exact partial evidence and limits recorded in their issue comments; #1198 remains open pending exact-head CI and required sole-owner source review. The umbrella acceptance is still incomplete.
grove commentedon Oct 29, 2026
Update: #1219 is now closed after exact-main DVM deep-tier run 40866922269 passed the 10-seed × 12-cycle histories; its 10 traces and 21/21 extended-history suite are retained in the run artifact. #1220, #1217 and #1222 are also closed. #1218, #1221, #1223 and #1224 remain open with their bounded implementation evidence and explicit limits recorded; #1198 remains open on the React 0.108.4 admission gate and pending owner review. The umbrella acceptance remains incomplete.
grove commentedon Oct 29, 2026
Update: #1218 is now closed after its bounded acceptance was verified on merged main candidate PR #1232 (run 40853118868). Exact numeric-scale observation, declared negative-control failures and setup rejection, the 12-case trigger/WAL × FULL/DIFFERENTIAL route witness, four SQL semantic mutants, and the installed receipt-prefix control passed; case and candidate evidence are retained in run artifacts.
#1219, #1220, #1217 and #1222 are also closed. #1221, #1223 and #1224 remain open with broader proof/qualification gaps recorded. #1198 remains open: PR #1199 has green exact-head CI, but the separate React 0.108.4 admission audit fails and the required sole-owner review is absent. Umbrella acceptance remains incomplete.
grove commentedon Oct 29, 2026
Update: #1218 and #1224 are now closed against merged main candidate PR #1232 after all bounded acceptance criteria passed in exact CI run 40853118868. Their comments link the semantic-control, identity-frame proof, and candidate-binding artifacts and state the remaining scope limits.
#1219, #1220, #1217 and #1222 are also closed. #1221 and #1223 remain open with explicit proof and runtime qualification gaps. #1198 remains open: PR #1199 has green exact-head CI, but its separate React 0.108.4 admission audit fails and the required sole-owner review is absent. Umbrella acceptance remains incomplete.
grove commentedon Oct 29, 2026
Current delivery status
The new PRs close only their documented bounded slices. #1221 and #1223 remain open for the acceptance recorded in their latest comments, so this umbrella issue remains open as well.
grove commentedon Oct 29, 2026
CI update for candidate PRs
Release evidence runtime controls, failed in the existingtest_dvm_failpoint_preserves_last_committed_result_and_recovers: CI observedeffective_refresh_mode=NO_DATAwhere the assertion expectedDIFFERENTIAL. Its scenario contains no TRUNCATE and does not exercise the new compaction guard. The same focused test passed twice locally, so a scheduler timing race is suspected but not yet confirmed. The parent workflow still has Unsafe ASan, Sensitive E2E 2/2, and packaged LSN runtime jobs in progress; I will rerun the failed job when the workflow completes.No PR has been merged. The #1198 compatibility admission and checksum remain unchanged pending the requested authorization.
grove commentedon Oct 29, 2026
CI update for candidate PRs
Sensitive E2E gate (1/2)failed in the existingtest_sum_case_differential_inserts: after the first insert refresh, the stream table held 20 while the defining query returned 50. The exact focused test passed locally, and the complete eight-test SUM CASE file passed on the same candidate. This points to a possible full-shard timing issue, but causality is not established.Release evidence runtime controlsfailpoint test also passed twice locally; its CI failure remains unexplained. The packaged checked-LSN runtime job passed.Sensitive E2E gate (2/2)is still running; I will retry only failed jobs once the parent workflow completes.grove commentedon Oct 29, 2026
Candidate CI update
refresh_st(), which can return after a lock-busy notice; commit06f6d584switches the SUM CASE scenarios to the existing convergence helper. All eight tests pass locally on the updated fix(dvm): correct nested join delta corrections (#1223) #1287 package; new exact-head CI is pending.grove commentedon Oct 29, 2026
Delivery update
c0b06387after 17/17 CI jobs passed on the final head.9e07d049after 26 checks passed and GitHub confirmed a clean merge state.grove commentedon Oct 29, 2026
Progress update: #1221 now has a bounded follow-up in PR #1293 at candidate 89ea301. The shared compaction eligibility decision is used by base and ST buffers, with local unit/lint/verification and 19 targeted E2E tests passing. GitHub CI is queued; full history and protected-window proof obligations remain open. #1221 and #1223 are still open, and previously excluded mutation issues remain excluded.
grove commentedon Oct 29, 2026
Additional scoped progress: #1292 now has a design-only PR, #1295, at candidate 980fe38. It records a source-traced design and preserves the INTERSECT admission guard until implementation and qualification. #1292 remains open; CI is queued. Existing #1221 and #1223 work also remains partial, and the excluded mutation issues remain out of scope.
grove commentedon Oct 31, 2026
Updated linked-issue status — 2026-10-31
b21a8e5946ed8892e467c975a88bdb89c17fd86fadds the checked signed image-event coefficients and repeated-image SQL case. Exact-head CI currently has 14 passed, 5 pending and no failures. This does not prove complete history equivalence, PostgreSQL aggregate semantics, transaction/source-incarnation transitions or all protected consumer windows.The umbrella remains open while #1221's broader acceptance and any other outstanding linked criteria remain unresolved. The mutation issues remain excluded.
grove commentedon Oct 31, 2026
Updated linked-issue status — 2026-10-31
47a93f89is already included in merged PR fix(dvm): fall back for unsafe nested projections #1303.9fc6b3941466810a4b073b5e05dd177f6a85ea87, with signed image-event coefficients, a repeated-image SQL case, and two scheduler E2E stability repairs prompted by exact-head CI. The affected local suites pass (45 tests), and exact-head CI is still running.joint-audit: the pinned pg-MDM revision lacks the runtime-epoch API required by the pg-react missing-receipt fixture. Its guard and MDM source remain unchanged.The umbrella remains open while #1221 and #1198 remain incomplete. The mutation issues remain excluded.
12 remaining items
grove commentedon Nov 2, 2026
Linked issue update — 2026-11-02
2dc63cd40278295ad50ece7b037c59c88228c6c4. Its exact-head S1 qualification, secret scan, upgrade completeness, checked-LSN evidence, and aggregate PR correctness checks all passed. The spec now records the installed-trigger domain case and finite nine-pair SQL image oracle.grove commentedon Nov 2, 2026
Linked issue update — 2026-11-02
a8782e1b446f94db3a531e1d0d18d492dbed4333, including integration, unit, Windows compile, shipping-image upgrade, sensitive E2E, light E2E, checked-LSN and aggregate PR correctness checks.grove commentedon Nov 2, 2026
Linked issue update — main CI for #1320
2dc63cd40278295ad50ece7b037c59c88228c6c4, including checked-LSN and upgrade completeness.grove commentedon Nov 2, 2026
Bounded slice merged: exhaustive two-identity SQL oracle
PR #1321 merged as
ff563398877cc1dd0087cb972ee75a2a02902ac1from exact head037afa6e49bade1778bea3426fb4611b996f6733. It adds a PostgreSQL execution oracle for all 81 initial/final state pairs across two row identities sharing a visible key, comparing productionkeyed_weight_sqlwith an independent complete-image bag model. It also makes installation-attestation writes locked and atomic after exact-head CI exposed a concurrent append race.Exact-head CI run 41462986844 and S1 qualification run 41462986991 passed. The post-merge main CI run 41471328036 is queued.
This remains a bounded test slice. Broader valid transaction histories, complete PostgreSQL capture/reduction equivalence, source-incarnation transitions, and proof/enforcement for every protected consumer window remain open. Keep #1221 and this umbrella issue open.
grove commentedon Nov 2, 2026
Next bounded slice: SQL NULL as a complete row image
PR #1322, exact head
071ca5d679c6de6d3809c07f3274e48407ceb526, adds a productionkeyed_weight_sqlexecution case for two identities sharing a visible key. One history transitions from a NULL payload to a non-NULL endpoint; the other ends with a NULL payload. Localjust fmt,just lint, and the focused PostgreSQL 18 Testcontainers case passed. Exact-head CI has started and is queued/running.This distinguishes a NULL-valued row image from an absent row. General transaction histories, complete capture/reduction equivalence, and all protected consumer-window obligations remain open; #1221 and #1225 stay open.
grove commentedon Nov 2, 2026
Progress update: PR #1322 was squash-merged as
b9d7224(2026-11-02). It adds a focused PostgreSQL oracle case for NULL-valued complete-image histories. On exact head071ca5d, the selected unit, integration, light-E2E, sensitive-E2E, shipping-upgrade, release-runtime, and checked-LSN checks all passed (17 passed, 0 failed). GitHub had left only thePR correctnessaggregation job queued when the authorized merge proceeded; it did not report a failure. Post-merge main CI run #41482843356 is queued on merge commitb9d7224. This remains one narrow #1221 boundary case; the wider #1225 acceptance work stays open.grove commentedon Nov 2, 2026
Linked issue update — source-incarnation runtime history
PR #1323, exact head
73a6c6d6cf740b0f8b64b87df743838b6710ca70, adds an installed PostgreSQL 18 regression for pending old-source images across same-name source recreation, in both row and statement trigger modes.just fmt,just lint,git diff --check, and the focused light-E2E Testcontainers run passed (1 passed, 18 skipped); hosted checks are queued.This is a bounded #1221 runtime case. The linked issue remains open for general SQL capture/reduction and protected consumer-window equivalence.
grove commentedon Nov 2, 2026
Main CI follow-up — #1322
Main CI run 41482843356 completed successfully on merge commit
b9d7224bca6d29e66eed185c5963ec2a67f003e4. #1221 and this umbrella issue remain open for the broader acceptance criteria.grove commentedon Nov 2, 2026
PostgreSQL qualification: interleaved complete-image histories
PR #1324, exact head
a7f23ac775ebca8f6a5f2bd257c170368dd18d79, adds an installed PostgreSQL 18 test for two source identities whose histories interleave and share complete output images. It runs in row and statement trigger modes, uses three separately committed multi-row updates, omits the source key from the output, observes numeric scale, forces differential maintenance with compaction eligible, and compares the resulting bag to the defining query.just fmt,just lint, and the focused Testcontainers case passed (1 passed, 26 skipped, PostgreSQL 18.6). Exact-head hosted checks are running. This is finite SQL/runtime evidence for one multi-identity history; arbitrary history reduction and all protected-window obligations remain open.grove commentedon Nov 2, 2026
Post-merge main CI — PR #1323
Main CI run 41496272241 completed successfully on exact merge commit
938f404108dd08b2b981f12d399c18430d63230a. Lint, coverage and secret scanning also passed on that commit. This qualifies the bounded source-incarnation runtime test; #1221 remains open for broader CDC history and protected-window acceptance.grove commentedon Nov 2, 2026
Merge status — PR #1324
PR #1324 merged on exact head
595448d90f7d655d0a43579d91093f985fb463f7as77305dae1f83b6160861c548dd6ce373140e47a8. All exact-head PR checks passed, including the packaged candidate-binding runtime case and all three light-E2E shards. Post-merge main CI run 41509109302 is queued. This adds finite PostgreSQL runtime evidence for multi-identity interleaved complete-image histories; arbitrary production normalization and all protected consumer-window obligations remain open.grove commentedon Nov 2, 2026
Protected consumer cuts — PR #1325
PR #1325 adds an installed PostgreSQL 18 test for two source identities whose complete output images interleave across an intermediate slow-consumer cut. It checks both row and statement CDC modes, exact bags at the fast and slow cuts, retained images after the fast consumer advances, and differential slow-consumer catch-up.
Local
just fmt,just lint, the focused Testcontainers case (1 passed, 27 skipped), and candidate-binding tests (10 passed) succeeded. Exact-head hosted checks are queued. This remains finite runtime evidence; arbitrary production history normalization and all protected-window schedules remain unproved.grove commentedon Nov 2, 2026
Candidate runtime qualification — PR #1325
On exact head
78bb83100b9dd4b04ab5ff552928365af1df3323, theRun packaged CDC image policy E2E and bind installed librarystep passed in CI run 41511322193. This step runs the new interleaved protected-cuts case against the packaged PostgreSQL extension and binds its installed library. Other exact-head light-E2E, release-runtime, and sensitive gates are still running. This is finite runtime evidence, not proof of all protected-window schedules or arbitrary normalization histories.grove commentedon Nov 2, 2026
Post-merge main CI — PR #1324
Main CI run 41509109302 completed successfully on exact merge commit
77305dae1f83b6160861c548dd6ce373140e47a8. PR #1324 remains bounded runtime evidence for multi-identity interleaved image histories; #1221 stays open for arbitrary production normalization and all protected-window obligations.grove commentedon Nov 2, 2026
Merge status — PR #1325
PR #1325 merged from exact head
78bb83100b9dd4b04ab5ff552928365af1df3323asc7288f557b03186e2afc4747d10e06acfc21b427. All exact-head checks passed: candidate CDC installed-library binding, all three light-E2E shards, both sensitive E2E gates, unit/integration, Windows, ASan, semantic mutants, release controls, and the final correctness gate. Post-merge main CI run 41519451349 is queued. #1221 remains open for arbitrary normalization proofs and broader protected-window coverage.