Skip to content

fix(sqlite-persistence): preserve expression index use - #1867

Open
KyleAMathews wants to merge 6 commits into
mainfrom
rfc-1659-ws5a-expression-index-oracle
Open

KyleAMathews wants to merge 6 commits into
mainfrom
rfc-1659-ws5a-expression-index-oracle

Conversation

@KyleAMathews

@KyleAMathews KyleAMathews commented Sep 20, 2026

Copy link
Copy Markdown
Collaborator

Compile validated JSON reference paths as canonical SQLite literals so runtime
predicates match persisted expression indexes. Queries keep returning the same
rows, but SQLite can now use the intended index instead of scanning the
collection table.

🎯 Changes

Root cause

Persisted index DDL embedded the JSON path as a SQL literal, while runtime
reference filters compiled the same path as a bound parameter. SQLite requires
an indexed expression to match the predicate expression structurally;
json_extract(value, ?) is semantically correct but cannot use an index defined
on json_extract(value, '$.path').

Approach

  • Render the already-validated base, type-tag, and tagged-value JSON paths with
    the existing SQLite literal helper.
  • Keep comparison values bound as parameters.
  • Add a real Better SQLite3 driver oracle that captures production predicate
    SQL, independently checks rows, and replays the query through
    EXPLAIN QUERY PLAN to require the named expression index.
  • Register and document the focused oracle campaign.

Key invariants

  • JSON path segments are validated before literalization.
  • User comparison values remain bound; only the compiler-owned path shape is
    literalized.
  • Runtime predicates and persisted index DDL use the same serialized ref
    expression for plain and tagged values.
  • Correct rows and named-index use are asserted separately to prevent row-only
    false greens.

Non-goals

  • This does not broaden raw-SQL support or change
    null/date/bigint/tagged-value coercion semantics.
  • Native-host execution and result ordering remain outside this oracle.
  • Transient RFC review and RED/GREEN evidence stays outside the product PR.

Trade-offs

JSON path shapes become part of the generated SQL text rather than path
bindings. That is required for SQLite expression-index compatibility; scalar
filter values remain parameterized.

✅ Checklist

  • I have tested this code locally with the focused commands below.

🚀 Release Impact

  • This change affects published code, and I have generated a changeset.
  • This change is docs/CI/dev-only (no release).

Verification

pnpm --filter @tanstack/db-sqlite-persistence-core test
pnpm --filter @tanstack/node-db-sqlite-persistence test
pnpm exec tsc -p packages/db-sqlite-persistence-core/tsconfig.json --noEmit
pnpm exec tsc -p packages/node-db-sqlite-persistence/tsconfig.json --noEmit
pnpm --filter @tanstack/db-sqlite-persistence-core build
pnpm --filter @tanstack/node-db-sqlite-persistence build

After merging current main, local results are: 96 SQLite persistence-core
tests passed; all 45 Node persistence tests passed, including all 3
expression-index oracle tests; both package TypeScript checks and builds
passed; and ESLint, Prettier, the commit hook, and diff validation passed.

Files changed

  • sqlite-core-adapter.ts: compile validated ref paths as canonical SQLite
    literals.
  • expression-index-oracle.test.ts: verify rows and the real named-index query
    plan across fixed and generated cases.
  • Package/docs metadata: expose and document the oracle campaign.
  • Changeset: publish the core behavior fix as a patch.

Part of #1659

Summary by CodeRabbit

  • Bug Fixes

    • Improved SQLite query planning for persisted expression indexes by standardizing JSON paths.
    • Nested JSON queries now correctly match corresponding indexes and avoid unnecessary table scans.
  • Documentation

    • Added guidance for running SQLite expression-index validation checks.
  • Tests

    • Added automated coverage for indexed query plans and accurate results across varied JSON paths and values.

@coderabbitai

coderabbitai Bot commented Sep 20, 2026

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 899a38a8-117f-4c84-860d-123e7a0c53a6

📥 Commits

Reviewing files that changed from the base of the PR and between 6db8fe7 and 28bbddd.

📒 Files selected for processing (1)
  • packages/node-db-sqlite-persistence/tests/expression-index-oracle.test.ts
🚧 Files skipped from review as they are similar to previous changes (1)
  • packages/node-db-sqlite-persistence/tests/expression-index-oracle.test.ts

Included review availability: Your plan provides up to 8 included reviews per hour; 2 remain after this review.


📝 Walkthrough

Walkthrough

compileRefExpressionSql now emits JSON paths as SQL literals so runtime predicates match persisted SQLite expression indexes. A property-based oracle verifies query results and index usage through EXPLAIN QUERY PLAN.

Changes

SQLite expression-index planning

Layer / File(s) Summary
Inline reference paths
packages/db-sqlite-persistence-core/src/sqlite-core-adapter.ts, .changeset/fix-sqlite-expression-index-planning.md
compileRefExpressionSql emits validated JSON paths as SQL literals and removes the four path bindings. A patch changeset records the fix.
Validate expression-index usage
packages/node-db-sqlite-persistence/tests/expression-index-oracle.test.ts, packages/node-db-sqlite-persistence/package.json, docs/contributing/oracle-coverage.md
The property-based oracle generates nested paths and scalar values, compares results with an independent scan, verifies named-index usage through EXPLAIN QUERY PLAN, and covers quoted identifiers, rejected DDL bindings, and prefix collisions. A package script and coverage documentation expose the oracle.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Bug fix

Merge Risk: ⚪ Minimal · up to 28bbd

The adapter emits canonical SQL literals for persisted JSON reference paths while comparison values remain bound, and the reported validation passes with no actionable merge-blocking risk evidenced.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 14 functions across 2 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and concisely describes the main change: preserving SQLite expression-index use for SQLite persistence.
Description check ✅ Passed The description follows the required template. It explains the changes and motivation, marks the testing and changeset checklist items, and includes verification results, scope, non-goals, and trade-o…
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 2
📝 Generate docstrings 💡
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
🛠️ Fix failing CI checks 💡
  • Commit to this branch
  • Create a new PR

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

…ion-index-oracle

# Conflicts:
#	docs/contributing/oracle-coverage.md
@pkg-pr-new

pkg-pr-new Bot commented Sep 20, 2026

Copy link
Copy Markdown
More templates

@tanstack/angular-db

npm i https://pkg.pr.new/@tanstack/angular-db@1867

@tanstack/browser-db-sqlite-persistence

npm i https://pkg.pr.new/@tanstack/browser-db-sqlite-persistence@1867

@tanstack/capacitor-db-sqlite-persistence

npm i https://pkg.pr.new/@tanstack/capacitor-db-sqlite-persistence@1867

@tanstack/cloudflare-durable-objects-db-sqlite-persistence

npm i https://pkg.pr.new/@tanstack/cloudflare-durable-objects-db-sqlite-persistence@1867

@tanstack/db

npm i https://pkg.pr.new/@tanstack/db@1867

@tanstack/db-ivm

npm i https://pkg.pr.new/@tanstack/db-ivm@1867

@tanstack/db-sqlite-persistence-core

npm i https://pkg.pr.new/@tanstack/db-sqlite-persistence-core@1867

@tanstack/electric-db-collection

npm i https://pkg.pr.new/@tanstack/electric-db-collection@1867

@tanstack/electron-db-sqlite-persistence

npm i https://pkg.pr.new/@tanstack/electron-db-sqlite-persistence@1867

@tanstack/expo-db-sqlite-persistence

npm i https://pkg.pr.new/@tanstack/expo-db-sqlite-persistence@1867

@tanstack/node-db-sqlite-persistence

npm i https://pkg.pr.new/@tanstack/node-db-sqlite-persistence@1867

@tanstack/offline-transactions

npm i https://pkg.pr.new/@tanstack/offline-transactions@1867

@tanstack/powersync-db-collection

npm i https://pkg.pr.new/@tanstack/powersync-db-collection@1867

@tanstack/query-db-collection

npm i https://pkg.pr.new/@tanstack/query-db-collection@1867

@tanstack/react-db

npm i https://pkg.pr.new/@tanstack/react-db@1867

@tanstack/react-native-db-sqlite-persistence

npm i https://pkg.pr.new/@tanstack/react-native-db-sqlite-persistence@1867

@tanstack/react-router-with-db

npm i https://pkg.pr.new/@tanstack/react-router-with-db@1867

@tanstack/rxdb-db-collection

npm i https://pkg.pr.new/@tanstack/rxdb-db-collection@1867

@tanstack/solid-db

npm i https://pkg.pr.new/@tanstack/solid-db@1867

@tanstack/svelte-db

npm i https://pkg.pr.new/@tanstack/svelte-db@1867

@tanstack/tauri-db-sqlite-persistence

npm i https://pkg.pr.new/@tanstack/tauri-db-sqlite-persistence@1867

@tanstack/trailbase-db-collection

npm i https://pkg.pr.new/@tanstack/trailbase-db-collection@1867

@tanstack/vue-db

npm i https://pkg.pr.new/@tanstack/vue-db@1867

commit: 28bbddd

@github-actions

github-actions Bot commented Sep 20, 2026

Copy link
Copy Markdown
Contributor

Size Change: 0 B

Total Size: 165 kB

ℹ️ View Unchanged
Filename Size
packages/db/dist/esm/client.js 3.66 kB
packages/db/dist/esm/collection-options.js 236 B
packages/db/dist/esm/collection/change-events.js 1.44 kB
packages/db/dist/esm/collection/changes.js 2.4 kB
packages/db/dist/esm/collection/cleanup-queue.js 794 B
packages/db/dist/esm/collection/events.js 481 B
packages/db/dist/esm/collection/index.js 4.36 kB
packages/db/dist/esm/collection/indexes.js 1.99 kB
packages/db/dist/esm/collection/lifecycle.js 2.15 kB
packages/db/dist/esm/collection/mutations.js 2.61 kB
packages/db/dist/esm/collection/state.js 6.51 kB
packages/db/dist/esm/collection/subscription.js 8.73 kB
packages/db/dist/esm/collection/sync.js 4.63 kB
packages/db/dist/esm/collection/transaction-metadata.js 144 B
packages/db/dist/esm/deferred.js 207 B
packages/db/dist/esm/errors.js 5.26 kB
packages/db/dist/esm/event-emitter.js 964 B
packages/db/dist/esm/index.js 3.71 kB
packages/db/dist/esm/indexes/auto-index.js 829 B
packages/db/dist/esm/indexes/base-index.js 1.14 kB
packages/db/dist/esm/indexes/basic-index.js 2.07 kB
packages/db/dist/esm/indexes/btree-index.js 2.26 kB
packages/db/dist/esm/indexes/index-registry.js 820 B
packages/db/dist/esm/indexes/reverse-index.js 376 B
packages/db/dist/esm/live-query-adapter.js 318 B
packages/db/dist/esm/live-query-observer.js 3.69 kB
packages/db/dist/esm/live-query-options.js 702 B
packages/db/dist/esm/live-query-window-controller.js 4.36 kB
packages/db/dist/esm/local-only.js 989 B
packages/db/dist/esm/local-storage.js 2.17 kB
packages/db/dist/esm/optimistic-action.js 359 B
packages/db/dist/esm/paced-mutations.js 496 B
packages/db/dist/esm/proxy.js 3.32 kB
packages/db/dist/esm/query/builder/functions.js 1.47 kB
packages/db/dist/esm/query/builder/index.js 6.69 kB
packages/db/dist/esm/query/builder/query-ir.js 116 B
packages/db/dist/esm/query/builder/ref-proxy.js 1.24 kB
packages/db/dist/esm/query/compiler/evaluators.js 1.92 kB
packages/db/dist/esm/query/compiler/expressions.js 560 B
packages/db/dist/esm/query/compiler/group-by.js 4.13 kB
packages/db/dist/esm/query/compiler/index.js 9.06 kB
packages/db/dist/esm/query/compiler/joins.js 2.95 kB
packages/db/dist/esm/query/compiler/lazy-targets.js 1.1 kB
packages/db/dist/esm/query/compiler/order-by.js 1.91 kB
packages/db/dist/esm/query/compiler/parent-routes.js 319 B
packages/db/dist/esm/query/compiler/route-metadata.js 1.24 kB
packages/db/dist/esm/query/compiler/select.js 1.58 kB
packages/db/dist/esm/query/effect.js 4.6 kB
packages/db/dist/esm/query/equality-value-identity.js 591 B
packages/db/dist/esm/query/expression-helpers.js 1.43 kB
packages/db/dist/esm/query/ir-stable-identity.js 4.04 kB
packages/db/dist/esm/query/ir.js 1.59 kB
packages/db/dist/esm/query/live-query-collection.js 391 B
packages/db/dist/esm/query/live/bucket-facade-adapter.js 2.73 kB
packages/db/dist/esm/query/live/collection-config-builder.js 6.97 kB
packages/db/dist/esm/query/live/collection-registry.js 264 B
packages/db/dist/esm/query/live/collection-subscriber.js 2.26 kB
packages/db/dist/esm/query/live/internal.js 145 B
packages/db/dist/esm/query/live/materialized-pipeline.js 2.32 kB
packages/db/dist/esm/query/live/ordered-source-loader.js 3.14 kB
packages/db/dist/esm/query/live/subset-demand-controller.js 1.26 kB
packages/db/dist/esm/query/live/utils.js 1.14 kB
packages/db/dist/esm/query/optimizer.js 2.91 kB
packages/db/dist/esm/query/query-once.js 359 B
packages/db/dist/esm/query/runtime-reference-identity.js 572 B
packages/db/dist/esm/query/subset-dedupe.js 486 B
packages/db/dist/esm/scheduler.js 1.34 kB
packages/db/dist/esm/SortedMap.js 1.3 kB
packages/db/dist/esm/strategies/debounceStrategy.js 247 B
packages/db/dist/esm/strategies/queueStrategy.js 428 B
packages/db/dist/esm/strategies/throttleStrategy.js 246 B
packages/db/dist/esm/transactions.js 3.71 kB
packages/db/dist/esm/utils.js 1.08 kB
packages/db/dist/esm/utils/array-utils.js 270 B
packages/db/dist/esm/utils/browser-polyfills.js 304 B
packages/db/dist/esm/utils/btree.js 4.51 kB
packages/db/dist/esm/utils/callbacks.js 174 B
packages/db/dist/esm/utils/comparison.js 1.49 kB
packages/db/dist/esm/utils/cursor.js 676 B
packages/db/dist/esm/utils/error.js 167 B
packages/db/dist/esm/utils/get-or-create.js 155 B
packages/db/dist/esm/utils/index-optimization.js 2.42 kB
packages/db/dist/esm/utils/type-guards.js 230 B
packages/db/dist/esm/utils/uuid.js 449 B
packages/db/dist/esm/virtual-props.js 360 B

compressed-size-action::db-package-size

@github-actions

Copy link
Copy Markdown
Contributor

Size Change: 0 B

Total Size: 7.34 kB

ℹ️ View Unchanged
Filename Size
packages/react-db/dist/esm/DbProvider.js 317 B
packages/react-db/dist/esm/HydrationBoundary.js 263 B
packages/react-db/dist/esm/index.js 330 B
packages/react-db/dist/esm/live-query-internals.js 282 B
packages/react-db/dist/esm/useLiveInfiniteQuery.js 1.9 kB
packages/react-db/dist/esm/useLiveQuery.js 2.68 kB
packages/react-db/dist/esm/useLiveQueryEffect.js 355 B
packages/react-db/dist/esm/useLiveSuspenseQuery.js 812 B
packages/react-db/dist/esm/usePacedMutations.js 401 B

compressed-size-action::react-db-package-size

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant