Incremental Watched Queries

[stevensJourney]

Overview

Our current Watched query implementations emit results whenever a change to a dependant SQLite table occurs. The table changes might not affect the query result set, but we still query and emit a new result set for each table change. The result sets typically contain the same data, but these results are new Array/object references which will cause re-renders in certain frameworks like React.

This PR overhauls, improves and extends upon the existing watched query implementations by introducing incremental watched queries.

Incrementally Watched Queries can be constructed with varying behaviour. This PR introduces the concept of comparison and differential watched queries.

Comparison based queries behave similar to standard watched queries. These queries still query the SQLite DB under the hood on each dependant table change, but they compare the result set and only incrementally yield results if a change has been made. The latest query result is yielded as the result set.

Differential queries watch a SQL query and report detailed information on the changes between result sets. This gives additional information such as the added, removed, updated rows between result set changes.

Implementation

The logic required for incrementally watched queries requires additional computation and introduces additional complexity to the implementation. For these reasons a new concept of a WatchedQuery class is introduced, along with a new query method allows building a instances of WatchedQuerys via watch and differentialWatch methods.

// Create an instance of a WatchedQuery.
const listsQuery = powersync
  .query<EnhancedListRecord>({
    sql: /* sql */ `
      SELECT
        ${LISTS_TABLE}.*,
        COUNT(${TODOS_TABLE}.id) AS total_tasks,
        SUM(
          CASE
            WHEN ${TODOS_TABLE}.completed = true THEN 1
            ELSE 0
          END
        ) as completed_tasks
      FROM
        ${LISTS_TABLE}
        LEFT JOIN ${TODOS_TABLE} ON ${LISTS_TABLE}.id = ${TODOS_TABLE}.list_id
      GROUP BY
        ${LISTS_TABLE}.id;
    `
  })
  .differentialWatch();

The listsQuery is smart, it:

• Automatically reprocesses itself if the PowerSync schema has been updated with updateSchema
• Automatically closes itself when the PowerSync client has been closed.
• Allows for the query parameters to be updated after instantiation.
• Allows shared listening to state changes.

// The registerListener method can be used multiple times to listen for updates.
// The returned dispose function can be used to unsubscribe from the updates.
const disposeSubscriber = listsQuery.registerListener({
  onData: (data) => {
    // This callback will be called whenever the data changes.
    // The data is the result of the executor.
    console.log('Data updated:', data);
  },
  onStateChange: (state) => {
    // This callback will be called whenever the state changes.
    // The state contains metadata about the query, such as isFetching, isLoading, etc.
    console.log(
      'State changed:', 
      state.error, 
      state.isFetching, 
      state.isLoading, 
      state.data
    );
  },
  onError: (error) => {
    // This callback will be called if the query fails.
    console.error('Query error:', error);
  }
});

WatchedQuery instances retain the latest state in memory. Sharing WatchedQuery instances can be used to introduce caching and reduce the number of duplicate DB queries between components.

The incremental logic is customisable. Diff based queries can specify custom logic for performing diffs on the relevant data set. By default a JSON.stringify approach is used. Different data sets might have more optimal implementations.

const watch = powersync
  .query({
    sql: /* sql */ `
      SELECT
        *
      FROM
        assets
    `,
    mapper: (raw) => {
      return {
        id: raw.id as string,
        make: raw.make as string
      };
    }
  })
  .differentialWatch({
    differentiator: {
      identify: (item) => item.id,
      compareBy: (item) => JSON.stringify(item)
    }
  });

Updates to query parameters can be performed in a single place, affecting all subscribers.

watch.updateSettings({
      query: new GetAllQuery({ sql: `SELECT * FROM assets OFFSET ? LIMIT 100`, parameters: [newOffset] })
    });

Reactivity

The existing watch method and Reactivity packages have been updated to use incremental queries with differentiation defined as an opt-in feature (defaults to no changes).

powersync.watch(
  'select * from assets',
  [],
  {
    onResult: () => {
      // This callback will be called whenever the data changes.
      console.log('Data updated');
    }
  },
  {
    comparator: {
      checkEquality: (current, previous) => {
        // This comparator will only report updates if the data changes.
        return JSON.stringify(current) === JSON.stringify(previous);
      }
    }
  }
);

/// React hooks
const { data, isLoading, isFetching } = useQuery(`SELECT * FROM cats WHERE breed = 'tabby'`, [], {
      differentiator: {
            identify: (item) => item.id,
            compareBy: (item) => JSON.stringify(item)
      }
  })

New hooks have also been added to use shared WatchedQuery instances.

const assetsQuery = powersync.query({
/// ....
}).watch();

/// In the component
export const MyComponent = () => {
   // In React one could import the `assetsQuery` or create a context provider for various queries
   const { data } = useWatchedQuerySubscription(assetsQuery)
}

The Vue and React hooks packages have been updated to remove duplicate implementations of common watched query logic. Historically these packages relied on manually implementing watched queries based off the onChange API in order to cater for exposing additional state and custom query executors. The new WatchedQuery APIs now support all the hook packages' requirements - this effectively reduces the heavy lifting in reactivity packages.

The React Supabase Todolist demo has been updated with some best practices for reducing re-renders using comparison based incrementally watched queries.

Differential Queries

These watched queries report the changes between result sets. The data member of a WatchedQuery's state is of the form

export interface WatchedQueryDifferential<RowType> {
  readonly added: ReadonlyArray<Readonly<RowType>>;
  /**
   * The entire current result set.
   * Array item object references are preserved between updates if the item is unchanged.
   *
   * e.g. In the query
   * ```sql
   *  SELECT name, make FROM assets ORDER BY make ASC;
   * ```
   *
   * If a previous result set contains an item (A) `{name: 'pc', make: 'Cool PC'}` and
   * an update has been made which adds another item (B) to the result set (the item A is unchanged) - then
   * the updated result set will be contain the same object reference, to item A, as the previous result set.
   * This is regardless of the item A's position in the updated result set.
   */
  readonly all: ReadonlyArray<Readonly<RowType>>;
  readonly removed: ReadonlyArray<Readonly<RowType>>;
  readonly updated: ReadonlyArray<WatchedQueryRowDifferential<Readonly<RowType>>>;
  readonly unchanged: ReadonlyArray<Readonly<RowType>>;
}

A common use case for this is processing newly created items as they are added. The YJS React Supabase Text Collab demo has been updated to take advantage of this feature. Document updates are watched via a differential incremental query. New updates are passed to YJS for consolidation as they are synced - the application code no longer needs to track which items were already passed to YJS.

[changeset-bot]

🦋 Changeset detected

Latest commit: 43f24fd

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 9 packages

Name	Type
@powersync/common	Minor
@powersync/vue	Minor
@powersync/react	Minor
@powersync/web	Minor
@powersync/node	Patch
@powersync/op-sqlite	Patch
@powersync/react-native	Patch
@powersync/tanstack-react-query	Patch
@powersync/diagnostics-app	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[Chriztiaan]

Happy with the changeset, while massive. Well documented and structured.

[rkistner]

It's a big PR, so just giving some high-level comments to start with:

Internals & Hooks

I really like the refactoring to move more of the logic into the internals - it's nice that the hooks are now fairly thin wrappers around the internal methods.

incrementalWatch

The db.incrementalWatch API feels a little difficult to follow with the current structure. It feels like the API changes completely based on which mode you select, and should perhaps rather be two separate APIs?

Another option is to invert the API to start with the query part first, for example something like this:

db.createQuery(sql, params).differentialWatch(...)

This can avoid the explosion of new methods on the database itself, instead having the different methods on a new Query-type class. You could perhaps also have different methods for creating this from different sources, e.g. to create from a Kysely or Drizzle query.

Incremental comparison queries

I see the main benefit of these to avoid re-renders, to improve rendering performance. It would be good to see some benchmarks to validate that this is worth the additional complexity, that we're not just moving overhead from the rendering part to the query part.

I also wonder if we should unify comparitors and differentiators, always using the differentiator API (identify + compareBy instead of just compare). This way, you can do the re-rendering on an even more granular level. So if you have 1 row out of 1000 results that changed, you can keep the other 999 row objects from the previous results, avoiding further re-renders on those rows. Once again, we'd need some form of benchmarks to check that this is actually worth it.

[rkistner]

The name is wrong in this example.

[stevensJourney]

👍 Fixed

[rkistner]

Should remove these log statements (including the one above)

[stevensJourney]

🙏 Removed this

[stevensJourney]

Update:

I've updated the implementation to guard the new watch methods behind the new query and customQuery methods on AbstractPowerSyncDatabase, following Ralf's recommendation. The PR description now includes usage examples.

Comparison based queries are now accessed via db.query().watch(). This watched query method does not perform a full diff on result sets; instead, it can exit early as soon as a change is detected. Use this method when you don't need to know the specifics of what changed and don't need to maintain previous object references in the result set. For cases where a full diff and object reference preservation are required, use db.query(...).differentialWatch().

The DifferentialWatchedQuery has been improved to better maintain previous array object references, further reducing unnecessary renders during incremental updates.

The WatchedQuery implementation now explicitly defines WatchedQueryState as readonly, making all state members and data items immutable. The Vue hooks are still in version 0; their APIs have been updated to also convey state as readonly. The React hooks package is already at a stable v1 release, so changing the API to report readonly state would be a breaking change. Backwards compatibility is maintained by conditionally applying readonly qualifiers if a differentiator parameter is supplied to the hook. Existing user code should not be affected.

An automated React profiling benchmark suite has been added in packages/react/tests/profile.test.tsx to measure the impact of differential queries on render performance. This test validates how using React memoization together with differential queries can significantly reduce render times. According to the results, incremental updates to the list widget render 60–80% faster compared to standard query methods. For details on the testing methodology, refer to the test file.
TLDR: The test starts with an initial amount of list items being rendered. We then incrementally add new items while measuring the render time of the entire list widget. Differential queries coupled with React memoization allow the incremental renders to only render the newly added item widgets, while the standard query methods re-render the entire widget. The graph below illustrates the render time for a list widget with various query and memoization methods.

[rkistner]

editor_id is a uuid in the sql, so this will fail.

But storing the data as hex also looks weird - not sure if this function works at all?

[rkistner]

This shouldn't be removed - it is used to avoid writing synced updates back to the db again.

[rkistner]

Need this as the origin (see _storeUpdate below). Could alternatively potentially use a different value, e.g. a Symbol indicating the source is the db.

[rkistner]

I feel like the previous approach of using the Set was simpler, since it didn't require an additional field in the database to filter the updates. We could still use the Set to only filter out the local updates, or alternatively apply those updates again (should be a no-op to apply the same update again).

[rkistner]

When I tested this with only one tab open, I often get into a state where the initial "diff" is sent with every update, e.g. "all: 20 rows, added: 20 rows". Haven't debugged further to figure out what is going wrong.

[rkistner]

This is duplicated.

[rkistner]

I haven't confirmed with testing, but it feels like accessing the differential state on the query could easily lead to race conditions where updates may be missed. Specifically, there could potentially be a case where the state is processed twice before it is processed in the app, leading to the diff from the first update to be missing.

With the normal async listeners this may not be an issue if the query waits for all listeners to return before processing the next one, but React hooks for example may be called at any later time.

So I'm wondering if we could send the diff in the callback, rather than the state on the query?

React hooks may have bigger issues though, since we have no control over when/how many times they are called. I'm not sure if it is actually feasible to expose the diff over a react hook's state at all.