fix(browser-db-sqlite-persistence): fairly schedule cold hydrations (#1868)

* fix(browser-db-sqlite-persistence): fairly schedule cold hydrations

* fix(persistence): preserve unscheduled restart startup

* fix(persistence): scope collection reset reloads

* fix(persistence): scope sequence-gap recovery

* fix(persistence): scope committed reloads

* docs(test): explain persistence oracle contracts

* test(browser-sqlite): surface OPFS oracle startup errors

* test(persistence): clarify synchronous admission checkpoint

* docs(oracles): label fairness coverage limits

* fix(sqlite-persistence): preserve coordinator progress

* test(persistence): complete gap recovery fixture

* fix(sqlite-persistence): release coordinator retention state

* test(sqlite-persistence): strengthen fairness oracle campaigns

* fix(sqlite-persistence): harden mutation envelope deduplication

* fix(sqlite-persistence): release queued reloads and fence stale startup reads

* test: allow real SQLite restart oracle CI time

* fix(sqlite-persistence): keep pre-apply aborts local and restore adapter routing

* fix(sqlite-persistence): retain default adapter cache until release

Author: KyleAMathews

.changeset/fix-shared-sqlite-hydration-fairness.md

@@ -0,0 +1,7 @@
+---
+'@tanstack/db-sqlite-persistence-core': patch
+'@tanstack/browser-db-sqlite-persistence': patch
+'@tanstack/electron-db-sqlite-persistence': patch
+---
+
+Schedule complete SQLite hydration units fairly without holding coordinator work inside the local hydration scope. Fence stale startup rows after a coordinator reset. Preserve per-Collection leader adapter routing, mutation results across transport retries, terminal coordinator disposal, real-adapter restart order, and promise-discovered shared scheduling.

.github/workflows/e2e-tests.yml

@@ -74,6 +74,11 @@ jobs:
           cd examples/react/start-ssr-e2e
           pnpm exec playwright install --with-deps chromium
 
+      - name: Run Browser SQLite OPFS fairness E2E tests
+        run: |
+          cd packages/browser-db-sqlite-persistence
+          pnpm test:opfs-fairness
+
       - name: Run React Start SSR E2E tests
         run: |
           cd examples/react/start-ssr-e2e

docs/contributing/oracle-coverage.md

@@ -111,7 +111,7 @@ comment and the current API/architecture contract before extending its model.
 | Opaque backend pagination | [window oracle](https://github.com/TanStack/db/blob/main/packages/query-db-collection/tests/cursor-pagination.oracle.test.ts), [cache histories](https://github.com/TanStack/db/blob/main/packages/query-db-collection/tests/cursor-pagination.cache-oracle.test.ts), [cache publication](https://github.com/TanStack/db/blob/main/packages/query-db-collection/tests/cursor-pagination.publication-oracle.test.ts), [browser acquisition boundaries](https://github.com/TanStack/db/blob/main/packages/query-db-collection/tests/cursor-pagination.boundary-oracle.test.ts), [QueryCollection integration](https://github.com/TanStack/db/blob/main/packages/query-db-collection/tests/cursor-pagination.integration.test.ts) | Full filter/sort/slice reference, opaque token transport, actual Query cache expiry/invalidation/GC, forced refresh during growth, protocol failure publication/recovery, bounded slice work, nested cancellation/replacement, reader abort, browser retry defaults, manual-write cache isolation, and production window publications. Stable backend sequences; not snapshot guarantees for changing endpoints. Peek-ahead remains enabled. |
 | Electric and TrailBase | [Electric histories](https://github.com/TanStack/db/blob/main/packages/electric-db-collection/tests/electric-oracle.property.test.ts), [recovery histories](https://github.com/TanStack/db/blob/main/packages/electric-db-collection/tests/electric-recovery-oracle.test.ts), [held resume snapshots](https://github.com/TanStack/db/blob/main/packages/electric-db-collection/tests/electric-resume-snapshot-races.test.ts), [PostgreSQL semantics](https://github.com/TanStack/db/blob/main/packages/electric-db-collection/e2e/sql-predicate-semantics.e2e.test.ts), [TrailBase contract](https://github.com/TanStack/db/blob/main/packages/trailbase-db-collection/tests/ORACLE.md) | Installed SDK delivery/framing, independent predicates, exact subscription arguments, restart/reset lineage, held certification and durability races, source-order publication before durability, and late errors. The queued-presence property runs identical fixed/random generators plus isolated seed-and-path replay across insert, update, delete, and truncate callbacks. The recovery fixtures use a mocked ShapeStream; they do not establish live Electric-service framing or native persistence-host behavior. |
 | PowerSync | [tests](https://github.com/TanStack/db/tree/main/packages/powersync-db-collection/tests), `tests/correctness-oracle.test.ts` | Applied receipt positions crossed with held peers, native SQLite/SDK and cleanup evidence. Run the focused owner with the package's `test:oracles` command. A timeout mutant proves a progress failure, not every value assertion. |
-| SQLite persistence and native hosts | [persisted histories](https://github.com/TanStack/db/blob/main/packages/db-sqlite-persistence-core/tests/persisted.test.ts), [reset/resume histories](https://github.com/TanStack/db/blob/main/packages/db-sqlite-persistence-core/tests/sqlite-core-adapter.test.ts), [dual-adapter resume snapshots](https://github.com/TanStack/db/blob/main/packages/db-sqlite-persistence-core/tests/sqlite-resume-snapshot.test.ts), [Browser composed-owner histories](https://github.com/TanStack/db/blob/main/packages/browser-db-sqlite-persistence/tests/per-collection-coordinator-oracle.test.ts), [Browser coordinator RPC](https://github.com/TanStack/db/blob/main/packages/browser-db-sqlite-persistence/tests/browser-coordinator.test.ts), [driver contracts](https://github.com/TanStack/db/blob/main/packages/db-sqlite-persistence-core/tests/contracts/sqlite-driver-contract.ts), [Node shared-handle scheduling](https://github.com/TanStack/db/blob/main/packages/node-db-sqlite-persistence/tests/node-driver.test.ts), [OP-SQLite shared-handle scheduling](https://github.com/TanStack/db/blob/main/packages/react-native-db-sqlite-persistence/tests/op-sqlite-driver.test.ts), [browser OPFS lifecycle](https://github.com/TanStack/db/blob/main/packages/browser-db-sqlite-persistence/tests/opfs-page-lifecycle-oracle.test.ts), [worker diagnostics](https://github.com/TanStack/db/blob/main/packages/browser-db-sqlite-persistence/tests/opfs-worker-diagnostics-oracle.test.ts), [Electron IPC and composed owner](https://github.com/TanStack/db/blob/main/packages/electron-db-sqlite-persistence/tests/electron-ipc.test.ts), [113-law manifest](https://github.com/TanStack/db/blob/main/packages/db-collection-e2e/src/fixtures/persisted-conformance-manifest.ts) | Core cache/remote rejection/peer/reopen histories, atomic reset/resume lineage, key-set evidence, dual-adapter races, and exact driver results. Browser composes public source commits with per-collection elected-owner routing and covers the complete committed-transaction wire partition through deterministic Node transport seams. Remote-subset histories distinguish logical demand, physical acquisitions, exact acquisition leases, and released replay tombstones. Electron composes source commits with a per-collection renderer owner, IPC adapter, real SQLite, and reopen checks. Same-handle Node and OP-SQLite tests cover transaction admission. Controlled OPFS page/worker histories cover ownership and diagnostic-cause retention. The reset/resume owners use sqlite3 CLI and in-memory node:sqlite seams; they do not prove multi-process WAL, mobile/Tauri, or other native-device execution. Distinct database handles rely on SQLite lock admission rather than one in-process queue. React Native hosts without async-context propagation must use the transaction driver supplied to the callback for nested work. Fake workers and synthetic page events do not prove native handle release or real bfcache admission. The Browser composed seams are not real multi-context/OPFS-worker execution; the Electron harness is not an actual Electron process unless its explicit runtime-bridge mode runs. An ownerless elected node suppresses core routing, while a follower may route demand to the elected owner; host coordinators retry only classified transport or admission failures while demand remains retained. The manifest excludes progressive and move suites; registration and shim runs are not device execution. |
+| SQLite persistence and native hosts | [persisted histories](https://github.com/TanStack/db/blob/main/packages/db-sqlite-persistence-core/tests/persisted.test.ts), [reset/resume histories](https://github.com/TanStack/db/blob/main/packages/db-sqlite-persistence-core/tests/sqlite-core-adapter.test.ts), [dual-adapter resume snapshots](https://github.com/TanStack/db/blob/main/packages/db-sqlite-persistence-core/tests/sqlite-resume-snapshot.test.ts), [Browser composed-owner histories](https://github.com/TanStack/db/blob/main/packages/browser-db-sqlite-persistence/tests/per-collection-coordinator-oracle.test.ts), [Browser coordinator RPC](https://github.com/TanStack/db/blob/main/packages/browser-db-sqlite-persistence/tests/browser-coordinator.test.ts), [shared-driver fairness](https://github.com/TanStack/db/blob/main/packages/browser-db-sqlite-persistence/tests/shared-driver-fairness-oracle.test.ts), [driver contracts](https://github.com/TanStack/db/blob/main/packages/db-sqlite-persistence-core/tests/contracts/sqlite-driver-contract.ts), [Node shared-handle scheduling](https://github.com/TanStack/db/blob/main/packages/node-db-sqlite-persistence/tests/node-driver.test.ts), [OP-SQLite shared-handle scheduling](https://github.com/TanStack/db/blob/main/packages/react-native-db-sqlite-persistence/tests/op-sqlite-driver.test.ts), [browser OPFS lifecycle](https://github.com/TanStack/db/blob/main/packages/browser-db-sqlite-persistence/tests/opfs-page-lifecycle-oracle.test.ts), [worker diagnostics](https://github.com/TanStack/db/blob/main/packages/browser-db-sqlite-persistence/tests/opfs-worker-diagnostics-oracle.test.ts), [Electron IPC and composed owner](https://github.com/TanStack/db/blob/main/packages/electron-db-sqlite-persistence/tests/electron-ipc.test.ts), [113-law manifest](https://github.com/TanStack/db/blob/main/packages/db-collection-e2e/src/fixtures/persisted-conformance-manifest.ts) | Core cache/remote rejection/peer/reopen histories, atomic reset/resume lineage, key-set evidence, dual-adapter races, queued coordinator reload/gap notifications during startup, unscheduled startup/reset overlap, and exact driver results. Browser composes public source commits with per-collection elected-owner routing and covers the complete committed-transaction wire partition through deterministic Node transport seams. Remote-subset histories distinguish logical demand, physical acquisitions, exact acquisition leases, and released replay tombstones. Electron composes source commits with a per-collection renderer owner, IPC adapter, real SQLite, and reopen checks. The shared-driver fairness owner checks K=1 complete-logical-hydrate scheduling with identical fixed, random, and seed-plus-path campaigns. It records public rows, raw dequeue reach, and a persist-first FIFO hostile control. Its Chromium OPFS fixture refines the provider boundary but does not establish a browser matrix, elapsed-time latency, unbounded eventuality, or multi-process coordination. Same-handle Node and OP-SQLite tests cover transaction admission. Controlled OPFS page/worker histories cover ownership and diagnostic-cause retention. The reset/resume owners use sqlite3 CLI and in-memory node:sqlite seams; they do not prove multi-process WAL, mobile/Tauri, or other native-device execution. Distinct database handles rely on SQLite lock admission rather than one in-process queue. React Native hosts without async-context propagation must use the transaction driver supplied to the callback for nested work. Fake workers and synthetic page events do not prove native handle release or real bfcache admission. The Browser composed seams are not real multi-context/OPFS-worker execution; the Electron harness is not an actual Electron process unless its explicit runtime-bridge mode runs. An ownerless elected node suppresses core routing, while a follower may route demand to the elected owner; host coordinators retry only classified transport or admission failures while demand remains retained. The manifest excludes progressive and move suites; registration and shim runs are not device execution. |
 | SQLite expression-index planning | [Node expression-index oracle](https://github.com/TanStack/db/blob/main/packages/node-db-sqlite-persistence/tests/expression-index-oracle.test.ts) | RFC #1659 invariant 8 owns identical persisted-index and runtime-expression shapes. Independent expected keys are checked against direct captured SQL, adapter results, and named-index plans. Generated BigInts use SQLite's signed range; one fixed case checks legacy oversized-value reads. Other limits: bounded unqualified JSON paths/scalars, Node BetterSQLite, and no null, arbitrary raw SQL, or native-host planning. Run the package's `test:oracles` campaign. |
 | Offline execution | [scheduler](https://github.com/TanStack/db/blob/main/packages/offline-transactions/tests/KeyScheduler.property.test.ts), [leadership](https://github.com/TanStack/db/blob/main/packages/offline-transactions/tests/leadership-replay.property.test.ts), [settlement](https://github.com/TanStack/db/blob/main/packages/offline-transactions/tests/transaction-settlement.property.test.ts), [serialization](https://github.com/TanStack/db/blob/main/packages/offline-transactions/tests/transaction-serializer.property.test.ts) | Declarative FIFO eligibility, per-transaction outcomes, durable state and typed wire trees. Issued work may finish after ownership loss, but new work must not start. Exactly-once network execution is not promised. |
 | Frameworks | [React conformance](https://github.com/TanStack/db/blob/main/packages/react-db/tests/conformance.test.tsx), [React pagination](https://github.com/TanStack/db/blob/main/packages/react-db/tests/infinite-query-conformance.test.tsx), [shared suites](https://github.com/TanStack/db/tree/main/packages/db-collection-e2e/src/suites) | Exact exposed rows/pages and each framework's own lifecycle cuts. A React witness does not prove Vue/Solid/Angular/Svelte scheduling. Preserve their receiving registrations. |

packages/browser-db-sqlite-persistence/e2e/shared-driver-fairness.opfs.html

@@ -0,0 +1,13 @@
+<!doctype html>
+<html lang="en">
+  <head>
+    <meta charset="UTF-8" />
+    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+    <link rel="icon" href="data:," />
+    <title>Shared driver OPFS fairness oracle</title>
+  </head>
+  <body>
+    <output id="oracle-status">running</output>
+    <script type="module" src="./shared-driver-fairness.opfs.ts"></script>
+  </body>
+</html>

packages/browser-db-sqlite-persistence/e2e/shared-driver-fairness.opfs.spec.ts

@@ -0,0 +1,102 @@
+/**
+ * Browser checkpoint assertions for the OPFS refinement. Expected public rows
+ * are rebuilt from the scenario IDs rather than from production output. The
+ * neutral case proves cold-query reach; the storm case requires no K=1
+ * violation. Failure-before-checkpoint, semantic mismatch, driver cleanup, and
+ * OPFS cleanup remain distinct outcomes so setup or teardown cannot satisfy the
+ * scheduling law.
+ */
+import { expect, test } from '@playwright/test'
+import type { Page } from '@playwright/test'
+import type { OPFSOracleResult } from './shared-driver-fairness.opfs'
+
+async function readOracleResult(
+  page: Page,
+  mode: `neutral` | `storm`,
+): Promise<OPFSOracleResult> {
+  let rejectPageError!: (error: Error) => void
+  const pageError = new Promise<never>((_resolve, reject) => {
+    rejectPageError = reject
+  })
+  void pageError.catch(() => undefined)
+  const onPageError = (error: Error) => {
+    rejectPageError(
+      new Error(
+        `OPFS fairness page failed before publishing a result: ${error.message}`,
+      ),
+    )
+  }
+  page.on(`pageerror`, onPageError)
+
+  try {
+    await page.goto(`/e2e/shared-driver-fairness.opfs.html?mode=${mode}`)
+    await Promise.race([
+      page.waitForFunction(
+        () => window.__tanstackDriverFairnessOracle !== undefined,
+      ),
+      pageError,
+    ])
+    return page.evaluate(() => window.__tanstackDriverFairnessOracle!)
+  } finally {
+    page.off(`pageerror`, onPageError)
+  }
+}
+
+function expectedHydratedCollections(scenarioId: string, count: number) {
+  return Array.from({ length: count }, (_, index) => ({
+    collectionId: `${scenarioId}-hydrate-${index}`,
+    rows: [
+      { id: `row-${index}-0`, value: index * 10 },
+      { id: `row-${index}-1`, value: index * 10 + 1 },
+    ],
+  }))
+}
+
+test(`real Chromium OPFS fixture reaches and cleans up cold hydration`, async ({
+  page,
+}) => {
+  const result = await readOracleResult(page, `neutral`)
+
+  if (result.status !== `complete`) throw new Error(result.primaryFailure)
+  expect(result.provider).toBe(`Chromium OPFSCoopSyncVFS worker`)
+  expect(result.observation.admittedHydrateIds).toHaveLength(2)
+  // These are actual public Collection rows captured after preload, compared
+  // with seed values built independently by this browser assertion.
+  expect(result.observation.hydratedCollections).toEqual(
+    expectedHydratedCollections(`opfs-neutral-reach`, 2),
+  )
+  expect(
+    result.observation.rawDequeues.some((entry) =>
+      entry.sql.startsWith(`SELECT key, value, metadata, row_version FROM`),
+    ),
+  ).toBe(true)
+  expect(result.observation.cleanupFailures).toEqual([])
+  expect(result.opfsCleanupFailures).toEqual([])
+})
+
+test(`real Chromium OPFS fixture bounds pending cold hydration behind persists`, async ({
+  page,
+}) => {
+  const result = await readOracleResult(page, `storm`)
+
+  if (result.status !== `complete`) throw new Error(result.primaryFailure)
+  expect(result.provider).toBe(`Chromium OPFSCoopSyncVFS worker`)
+  expect(result.observation.admittedHydrateIds).toHaveLength(4)
+  expect(result.observation.hydratedCollections).toEqual(
+    expectedHydratedCollections(`opfs-fixed-persist-storm`, 4),
+  )
+  // This is the semantic RED checkpoint. Setup, wall time, and cleanup are
+  // reported independently and cannot satisfy this assertion.
+  if (result.violation !== undefined) {
+    throw new Error(
+      `real OPFS fairness mismatch: ${JSON.stringify(result.violation)}; ` +
+        `logical completion order: ${JSON.stringify(result.observation.logicalCompletionOrder)}; ` +
+        `driver admissions: ${result.observation.driverAdmissions.length}; ` +
+        `raw dequeues: ${result.observation.rawDequeues.length}; ` +
+        `driver cleanup diagnostics: ${JSON.stringify(result.observation.cleanupFailures)}; ` +
+        `OPFS cleanup diagnostics: ${JSON.stringify(result.opfsCleanupFailures)}`,
+    )
+  }
+  expect(result.observation.cleanupFailures).toEqual([])
+  expect(result.opfsCleanupFailures).toEqual([])
+})