From 37272c9e314f314cfab34d81833bb0a11f2dbc21 Mon Sep 17 00:00:00 2001 From: benitav Date: Wed, 9 Jul 2025 12:57:55 +0200 Subject: [PATCH 01/15] init --- docs.json | 1 + usage/use-case-examples.mdx | 3 +- usage/use-case-examples/raw-tables.mdx | 74 ++++++++++++++++++++++++++ 3 files changed, 76 insertions(+), 2 deletions(-) create mode 100644 usage/use-case-examples/raw-tables.mdx diff --git a/docs.json b/docs.json index b5784660..9f110723 100644 --- a/docs.json +++ b/docs.json @@ -187,6 +187,7 @@ "usage/use-case-examples/offline-only-usage", "usage/use-case-examples/postgis", "usage/use-case-examples/prioritized-sync", + "usage/use-case-examples/raw-tables", "usage/use-case-examples/custom-write-checkpoints" ] }, diff --git a/usage/use-case-examples.mdx b/usage/use-case-examples.mdx index 403405f1..c5e66958 100644 --- a/usage/use-case-examples.mdx +++ b/usage/use-case-examples.mdx @@ -1,8 +1,6 @@ --- title: "Use Case Examples" description: "Learn how to use PowerSync in common use cases" -mode: wide -sidebarTitle: Overview --- The following examples are available to help you get started with specific use cases for PowerSync: @@ -19,6 +17,7 @@ The following examples are available to help you get started with specific use c + ## Additional Resources diff --git a/usage/use-case-examples/raw-tables.mdx b/usage/use-case-examples/raw-tables.mdx new file mode 100644 index 00000000..e9cc292a --- /dev/null +++ b/usage/use-case-examples/raw-tables.mdx @@ -0,0 +1,74 @@ +--- +title: "Raw SQLite Tables to bypass JSON View Limitations" +description: "Use raw tables for native SQLite functionality and improved performance." +sidebarTitle: "Raw SQLite Tables" +--- + + +Raw tables are an experimental feature for the purpose of getting feedback from users and refining the APIs. + + +Raw tables allow you to define native SQLite tables in the client-side schema, bypassing PowerSync's default [JSON-based view system](/architecture/client-architecture#schema). This eliminates overhead associated with extracting values from the JSON data and provides access to advanced SQLite features like foreign key constraints and custom indexes. + + +*Availability** + +Raw tables were introduced in the following versions of our client SDKs: +- TODO + + +## When to Use Raw Tables + +Consider raw tables when you need: + +- **Advanced SQLite features** like `FOREIGN KEY` and `ON DELETE CASCADE` constraints +- **Indexes** - PowerSync's default schema has limited support for indexes (todo: elaborate), while raw tables give you complete control to create indexes on expressions, use `GENERATED` columns, etc +- **Improved performance** for complex queries (e.g., `SELECT SUM(value) FROM transactions`) - raw tables more efficiently get these values directly from the SQLite column, instead of extracting the value from the JSON object on every row +- **Reduced storage overhead** - eliminate JSON object overhead for each row in `ps_data__.data` column +- **To manually create tables** - Sometimes you need full control over table creation, for example when implementing custom triggers + +## How Raw Tables Work + +### Current JSON-Based System + +Currently the sync system involves two general steps: + +1. Download sync bucket operations from the PowerSync Service +2. Once the client has a complete checkpoint and no local changes, sync the local database with the bucket operations + +The bucket operations use JSON to store the individual operation data. The local database uses tables with a simple schemaless `ps_data__` structure. + +PowerSync then creates views on that table that extract JSON fields to resemble standard tables. + +### Raw Tables Approach + +At a high level, raw tables consist of two components: + +1. **Schema definition** - Define the raw table in your schema with a table name used by the PowerSync Service so it knows to forward operations to your table instead of the default `ps_data__
` + +2. **Custom SQL statements** - SQL statements to log updates in the upload queue: + + **Upsert statement** - SQL statement to run for `PUT` operations, using prepared statement parameters that can reference the row ID or extract values from JSON data: + ```sql + todo simple example + ``` + + **Delete statement** - SQL statement to run for `REMOVE` operations: + ```sql + todo simple example + ``` + +For details, see the implementation guide below. + +## Implementation Guide + +todo + +## Support and Feedback + +Raw tables are an experimental feature. We're actively seeking feedback on: + +- API design and developer experience +- Additional features or optimizations needed + +Join our [Discord community](https://discord.gg/powersync) to share your experience and get help. \ No newline at end of file From 98ed080f29c65058669ded1a6423878f63a99776 Mon Sep 17 00:00:00 2001 From: Simon Binder Date: Wed, 9 Jul 2025 15:50:27 +0200 Subject: [PATCH 02/15] Code snippets --- usage/use-case-examples/raw-tables.mdx | 220 ++++++++++++++++++++++--- 1 file changed, 197 insertions(+), 23 deletions(-) diff --git a/usage/use-case-examples/raw-tables.mdx b/usage/use-case-examples/raw-tables.mdx index e9cc292a..782f660e 100644 --- a/usage/use-case-examples/raw-tables.mdx +++ b/usage/use-case-examples/raw-tables.mdx @@ -36,33 +36,207 @@ Currently the sync system involves two general steps: 1. Download sync bucket operations from the PowerSync Service 2. Once the client has a complete checkpoint and no local changes, sync the local database with the bucket operations -The bucket operations use JSON to store the individual operation data. The local database uses tables with a simple schemaless `ps_data__` structure. +The bucket operations use JSON to store the individual operation data. The local database uses tables with a simple schemaless `ps_data__` structure +containing only an `id` (TEXT) and `data` (JSON) column. -PowerSync then creates views on that table that extract JSON fields to resemble standard tables. +PowerSync automatically creates views on that table that extract JSON fields to resemble standard tables reflecting your schema. ### Raw Tables Approach -At a high level, raw tables consist of two components: - -1. **Schema definition** - Define the raw table in your schema with a table name used by the PowerSync Service so it knows to forward operations to your table instead of the default `ps_data__
` - -2. **Custom SQL statements** - SQL statements to log updates in the upload queue: - - **Upsert statement** - SQL statement to run for `PUT` operations, using prepared statement parameters that can reference the row ID or extract values from JSON data: - ```sql - todo simple example - ``` - - **Delete statement** - SQL statement to run for `REMOVE` operations: - ```sql - todo simple example - ``` - -For details, see the implementation guide below. - -## Implementation Guide - -todo +When opting in to raw tables, you are responsible for creating the tables before using them - PowerSync will no longer +create them automatically. + +Because PowerSync takes no control over raw tables, you need to manually: + +1. Tell PowerSync how to map the [schemaless protocol](/architecture/powersync-protocol#protocol) into your existing tables. +2. Configure custom triggers to forward local writes to PowerSync. + +For the purpose of this example, consider a simple table like this: + +```sql +CREATE TABLE todo_lists ( + id TEXT NOT NULL PRIMARY KEY, + created_by TEXT NOT NULL, + title TEXT NOT NULL, + content TEXT +) STRICT; +``` + +#### Syncing into raw tables + +To sync into the existing `todo_lists` table instead of `ps_data__`, PowerSync needs the SQL statements extracting +columns from the untyped JSON protocol used during syncing. +This involves specifying two SQL statements: + +1. A `put` SQL statement for upserts, responsible for creating a `todo_list` row or updating it based on its `id` and data columns. +2. A `delete` SQL statement responsible for deletions. + +The PowerSync client as part of our SDKs will automatically run these statements in response to sync lines being sent from +the PowerSync service. + +To reference the ID or extract values, prepared statements with parameters are used. `delete` statements can reference the id +of the affected row, while `put` statements can also reference individual column values. +Declaring these statements and parameters happens as part of the schema passed to PowerSync databases: + + + + +Raw tables are not included in the regular schema object. Instead, add them afterwards using `withRawTables`. +For each raw table, specify the `put` and `delete` statement. The values of parameters are described as a JSON +array either containing: + +- the string `Id` to reference the id of the affected row. +- the object `{ Column: name }` to reference the value of the column `name`. + +```JavaScript +const mySchema = new Schema({}); +customSchema.withRawTables({ + // The name here doesn't have to match the name of the table in SQL. Instead, it's used to match + // the table name from the backend database as sent by the PowerSync service. + todo_lists: { + put: { + sql: 'INSERT OR REPLACE INTO todo_lists (id, created_by, title, content) VALUES (?, ?, ?, ?)', + params: ['Id', { Column: 'created_by' }, { Column: 'title' }, { Column: 'content' }] + }, + delete: { + sql: 'DELETE FROM lists WHERE id = ?', + params: ['Id'] + } + } +}); +``` + +We will simplify this API after understanding the use-cases for raw tables better. + + + +Raw tables are not part of the regular tables list and can be defined with the optional `rawTables` parameter. + +```dart +final schema = Schema(const [], rawTables: const [ + RawTable( + // The name here doesn't have to match the name of the table in SQL. Instead, it's used to match + // the table name from the backend database as sent by the PowerSync service. + name: 'todo_lists', + put: PendingStatement( + sql: 'INSERT OR REPLACE INTO todo_lists (id, created_by, title, content) VALUES (?, ?, ?, ?)', + params: [ + PendingStatementValue.id(), + PendingStatementValue.column('created_by'), + PendingStatementValue.column('title'), + PendingStatementValue.column('content'), + ], + ), + delete: PendingStatement( + sql: 'DELETE FROM todo_lists WHERE id = ?', + params: [ + PendingStatementValue.id(), + ], + ), + ), +]); +``` + + + +To define a raw table, include it in the list of tables passed to the `Schema`: + +```Kotlin +val schema = Schema(listOf( + RawTable( + // The name here doesn't have to match the name of the table in SQL. Instead, it's used to match + // the table name from the backend database as sent by the PowerSync service. + name = "todo_lists", + put = PendingStatement( + "INSERT OR REPLACE INTO todo_lists (id, created_by, title, content) VALUES (?, ?, ?, ?)", + listOf( + PendingStatementParameter.Id, + PendingStatementParameter.Column("created_by"), + PendingStatementParameter.Column("title"), + PendingStatementParameter.Column("content") + ) + ), + delete = PendingStatement( + "DELETE FROM todo_lists WHERE id = ?", listOf(PendingStatementParameter.Id) + ) + ) +)) +``` + + + +Unfortunately, raw tables are not available in the Swift SDK yet. + + + +Unfortunately, raw tables are not available in the .NET SDK yet. + + + + +After adding raw tables to the schema, you're also responsible for creating them by executing the +corresponding `CREATE TABLE` statement before `connect()`-ing the database. + +Also note that raw tables are only supported by the new Rust sync client, which is currently opt-in. + +#### Collecting local writes on raw tables + +PowerSync uses an internal SQLite table to collect local writes. For PowerSync-managed views, a trigger for +insertions, updates and deletions automatically forwards local writes into this table. +When using raw tables, defining those triggers is your responsibility. + +The PowerSync SQLite extension creates an insert-only virtual table named `powersync_crud` with these columns: + +```SQL +CREATE TABLE pwoersync_crud( + -- The type of operation: 'PUT' or 'DELETE' + op TEXT, + -- The id of the affected row + id TEXT, + type TEXT, + -- optional (not set on deletes): The column values for the row + data TEXT, + -- optional: Previous column values to include in a CRUD entry + old_values TEXT, + -- optional: Metadata for the write to include in a CRUD entry + metadata TEXT, +); +``` + +The virtual table associates writes with the current transaction and ensures writes made during the sync +process (applying server-side changes) don't count as local writes. +This means that triggers can be defined on raw tables like so: + +```SQL +CREATE TRIGGER todo_lists_insert + AFTER INSERT ON todo_lists + FOR EACH ROW + BEGIN + INSERT INTO powersync_crud (op, id, type, data) VALUES ('PUT', NEW.id, 'todo_lists', json_object( + 'created_by', NEW.created_by, + 'title', NEW.title, + 'content', NEW.content + )); + END; + +CREATE TRIGGER todo_lists_update + AFTER INSERT ON todo_lists + FOR EACH ROW + BEGIN + INSERT INTO powersync_crud (op, id, type, data) VALUES ('PUT', NEW.id, 'todo_lists', json_object( + 'created_by', NEW.created_by, + 'title', NEW.title, + 'content', NEW.content + )); + END; + +CREATE TRIGGER todo_lists_delete + AFTER DELETE ON todo_lists + FOR EACH ROW + BEGIN + INSERT INTO powersync_crud (op, id, type) VALUES ('DELETE', OLD.id, 'todo_lists'); + END; +``` ## Support and Feedback From c6a1af8c85c2fba880ba5db808e86acca951cca1 Mon Sep 17 00:00:00 2001 From: Simon Binder Date: Wed, 9 Jul 2025 17:50:52 +0200 Subject: [PATCH 03/15] Fix JS typo --- usage/use-case-examples/raw-tables.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/usage/use-case-examples/raw-tables.mdx b/usage/use-case-examples/raw-tables.mdx index 782f660e..1c6fa3b0 100644 --- a/usage/use-case-examples/raw-tables.mdx +++ b/usage/use-case-examples/raw-tables.mdx @@ -90,7 +90,7 @@ array either containing: ```JavaScript const mySchema = new Schema({}); -customSchema.withRawTables({ +mySchema.withRawTables({ // The name here doesn't have to match the name of the table in SQL. Instead, it's used to match // the table name from the backend database as sent by the PowerSync service. todo_lists: { From c28ced2ff3d3a50bd40c80bc738caee130497f13 Mon Sep 17 00:00:00 2001 From: Simon Binder Date: Wed, 9 Jul 2025 17:54:15 +0200 Subject: [PATCH 04/15] More typos --- usage/use-case-examples/raw-tables.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/usage/use-case-examples/raw-tables.mdx b/usage/use-case-examples/raw-tables.mdx index 1c6fa3b0..29a2bac0 100644 --- a/usage/use-case-examples/raw-tables.mdx +++ b/usage/use-case-examples/raw-tables.mdx @@ -188,7 +188,7 @@ When using raw tables, defining those triggers is your responsibility. The PowerSync SQLite extension creates an insert-only virtual table named `powersync_crud` with these columns: ```SQL -CREATE TABLE pwoersync_crud( +CREATE TABLE powersync_crud( -- The type of operation: 'PUT' or 'DELETE' op TEXT, -- The id of the affected row From 7cc34df0b46ed87fd4b9630e91fba36d49bfa3a5 Mon Sep 17 00:00:00 2001 From: Simon Binder Date: Wed, 9 Jul 2025 17:57:38 +0200 Subject: [PATCH 05/15] Make it clearer that powersync_crud is virtual --- usage/use-case-examples/raw-tables.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/usage/use-case-examples/raw-tables.mdx b/usage/use-case-examples/raw-tables.mdx index 29a2bac0..81fdb93a 100644 --- a/usage/use-case-examples/raw-tables.mdx +++ b/usage/use-case-examples/raw-tables.mdx @@ -188,7 +188,7 @@ When using raw tables, defining those triggers is your responsibility. The PowerSync SQLite extension creates an insert-only virtual table named `powersync_crud` with these columns: ```SQL -CREATE TABLE powersync_crud( +CREATE VIRTUAL TABLE powersync_crud( -- The type of operation: 'PUT' or 'DELETE' op TEXT, -- The id of the affected row From 5bc8aefa6c388a7b5c068f4e60dbf7307c958565 Mon Sep 17 00:00:00 2001 From: benitav Date: Thu, 10 Jul 2025 14:01:33 +0200 Subject: [PATCH 06/15] polish --- usage/use-case-examples/raw-tables.mdx | 35 +++++++++++++------------- 1 file changed, 18 insertions(+), 17 deletions(-) diff --git a/usage/use-case-examples/raw-tables.mdx b/usage/use-case-examples/raw-tables.mdx index 81fdb93a..4e0b1221 100644 --- a/usage/use-case-examples/raw-tables.mdx +++ b/usage/use-case-examples/raw-tables.mdx @@ -8,13 +8,17 @@ sidebarTitle: "Raw SQLite Tables" Raw tables are an experimental feature for the purpose of getting feedback from users and refining the APIs. -Raw tables allow you to define native SQLite tables in the client-side schema, bypassing PowerSync's default [JSON-based view system](/architecture/client-architecture#schema). This eliminates overhead associated with extracting values from the JSON data and provides access to advanced SQLite features like foreign key constraints and custom indexes. +By default, PowerSync uses a [JSON-based view system](/architecture/client-architecture#schema). Raw tables allow you to define native SQLite tables in the client-side schema, bypassing this. + +This eliminates overhead associated with extracting values from the JSON data and provides access to advanced SQLite features like foreign key constraints and custom indexes. *Availability** Raw tables were introduced in the following versions of our client SDKs: - TODO + +Also note that raw tables are only supported by the new [Rust-based sync client](https://releases.powersync.com/announcements/improved-sync-performance-in-our-client-sdks), which is currently opt-in. ## When to Use Raw Tables @@ -34,21 +38,19 @@ Consider raw tables when you need: Currently the sync system involves two general steps: 1. Download sync bucket operations from the PowerSync Service -2. Once the client has a complete checkpoint and no local changes, sync the local database with the bucket operations +2. Once the client has a complete checkpoint and no pending local changes in the upload queue, sync the local database with the bucket operations -The bucket operations use JSON to store the individual operation data. The local database uses tables with a simple schemaless `ps_data__` structure -containing only an `id` (TEXT) and `data` (JSON) column. +The bucket operations use JSON to store the individual operation data. The local database uses tables with a simple schemaless `ps_data__` structure containing only an `id` (TEXT) and `data` (JSON) column. PowerSync automatically creates views on that table that extract JSON fields to resemble standard tables reflecting your schema. ### Raw Tables Approach -When opting in to raw tables, you are responsible for creating the tables before using them - PowerSync will no longer -create them automatically. +When opting in to raw tables, you are responsible for creating the tables before using them - PowerSync will no longer create them automatically. Because PowerSync takes no control over raw tables, you need to manually: -1. Tell PowerSync how to map the [schemaless protocol](/architecture/powersync-protocol#protocol) into your existing tables. +1. Tell PowerSync how to map the [schemaless protocol](/architecture/powersync-protocol#protocol) to your raw tables when syncing data. 2. Configure custom triggers to forward local writes to PowerSync. For the purpose of this example, consider a simple table like this: @@ -64,24 +66,22 @@ CREATE TABLE todo_lists ( #### Syncing into raw tables -To sync into the existing `todo_lists` table instead of `ps_data__`, PowerSync needs the SQL statements extracting +To sync into the raw `todo_lists` table instead of `ps_data__`, PowerSync needs the SQL statements extracting columns from the untyped JSON protocol used during syncing. This involves specifying two SQL statements: 1. A `put` SQL statement for upserts, responsible for creating a `todo_list` row or updating it based on its `id` and data columns. 2. A `delete` SQL statement responsible for deletions. -The PowerSync client as part of our SDKs will automatically run these statements in response to sync lines being sent from -the PowerSync service. +The PowerSync client as part of our SDKs will automatically run these statements in response to sync lines being sent from the PowerSync Service. -To reference the ID or extract values, prepared statements with parameters are used. `delete` statements can reference the id -of the affected row, while `put` statements can also reference individual column values. +To reference the ID or extract values, prepared statements with parameters are used. `delete` statements can reference the id of the affected row, while `put` statements can also reference individual column values. Declaring these statements and parameters happens as part of the schema passed to PowerSync databases: -Raw tables are not included in the regular schema object. Instead, add them afterwards using `withRawTables`. +Raw tables are not included in the regular `Schema()` object. Instead, add them afterwards using `withRawTables`. For each raw table, specify the `put` and `delete` statement. The values of parameters are described as a JSON array either containing: @@ -89,7 +89,10 @@ array either containing: - the object `{ Column: name }` to reference the value of the column `name`. ```JavaScript -const mySchema = new Schema({}); +const mySchema = new Schema({ + // Define your PowerSync-managed schema here + // ... +}); mySchema.withRawTables({ // The name here doesn't have to match the name of the table in SQL. Instead, it's used to match // the table name from the backend database as sent by the PowerSync service. @@ -177,15 +180,13 @@ Unfortunately, raw tables are not available in the .NET SDK yet. After adding raw tables to the schema, you're also responsible for creating them by executing the corresponding `CREATE TABLE` statement before `connect()`-ing the database. -Also note that raw tables are only supported by the new Rust sync client, which is currently opt-in. - #### Collecting local writes on raw tables PowerSync uses an internal SQLite table to collect local writes. For PowerSync-managed views, a trigger for insertions, updates and deletions automatically forwards local writes into this table. When using raw tables, defining those triggers is your responsibility. -The PowerSync SQLite extension creates an insert-only virtual table named `powersync_crud` with these columns: +The [PowerSync SQLite extension](https://github.com/powersync-ja/powersync-sqlite-core) creates an insert-only virtual table named `powersync_crud` with these columns: ```SQL CREATE VIRTUAL TABLE powersync_crud( From 18593e1eebc560c804c4aea791066a7fb7146bb2 Mon Sep 17 00:00:00 2001 From: benitav Date: Fri, 11 Jul 2025 13:15:25 +0200 Subject: [PATCH 07/15] polish --- usage/use-case-examples/raw-tables.mdx | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/usage/use-case-examples/raw-tables.mdx b/usage/use-case-examples/raw-tables.mdx index 4e0b1221..471a89be 100644 --- a/usage/use-case-examples/raw-tables.mdx +++ b/usage/use-case-examples/raw-tables.mdx @@ -26,7 +26,7 @@ Also note that raw tables are only supported by the new [Rust-based sync client] Consider raw tables when you need: - **Advanced SQLite features** like `FOREIGN KEY` and `ON DELETE CASCADE` constraints -- **Indexes** - PowerSync's default schema has limited support for indexes (todo: elaborate), while raw tables give you complete control to create indexes on expressions, use `GENERATED` columns, etc +- **Indexes** - PowerSync's default schema has basic support for indexes on columns, while raw tables give you complete control to create indexes on expressions, use `GENERATED` columns, etc - **Improved performance** for complex queries (e.g., `SELECT SUM(value) FROM transactions`) - raw tables more efficiently get these values directly from the SQLite column, instead of extracting the value from the JSON object on every row - **Reduced storage overhead** - eliminate JSON object overhead for each row in `ps_data__
.data` column - **To manually create tables** - Sometimes you need full control over table creation, for example when implementing custom triggers @@ -183,7 +183,7 @@ corresponding `CREATE TABLE` statement before `connect()`-ing the database. #### Collecting local writes on raw tables PowerSync uses an internal SQLite table to collect local writes. For PowerSync-managed views, a trigger for -insertions, updates and deletions automatically forwards local writes into this table. +insertions, updates and deletions automatically forwards local mutations into this table. When using raw tables, defining those triggers is your responsibility. The [PowerSync SQLite extension](https://github.com/powersync-ja/powersync-sqlite-core) creates an insert-only virtual table named `powersync_crud` with these columns: @@ -204,7 +204,7 @@ CREATE VIRTUAL TABLE powersync_crud( ); ``` -The virtual table associates writes with the current transaction and ensures writes made during the sync +The virtual table associates local mutations with the current transaction and ensures writes made during the sync process (applying server-side changes) don't count as local writes. This means that triggers can be defined on raw tables like so: @@ -238,6 +238,9 @@ CREATE TRIGGER todo_lists_delete INSERT INTO powersync_crud (op, id, type) VALUES ('DELETE', OLD.id, 'todo_lists'); END; ``` +## Migrations + +In PowerSync's [JSON-based view system](/architecture/client-architecture#schema) the client-side schema is applied to the schemaless data, meaning no migrations are required. Raw tables however are excluded from this, so it is the developers responsibility to manage migrations for these tables. ## Support and Feedback From 1f0a84092725d32199e8934d7dfed4672c98bb3f Mon Sep 17 00:00:00 2001 From: benitav Date: Fri, 11 Jul 2025 13:22:33 +0200 Subject: [PATCH 08/15] mention raw tables in client architecture --- architecture/client-architecture.mdx | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/architecture/client-architecture.mdx b/architecture/client-architecture.mdx index 0cf51c49..e5f41368 100644 --- a/architecture/client-architecture.mdx +++ b/architecture/client-architecture.mdx @@ -64,4 +64,8 @@ The client SDK maintains the following tables: Most rows will be present in at least two tables — the `ps_data__
` table, and in `ps_oplog`. It may be present multiple times in `ps_oplog`, if it was synced via multiple buckets. -The copy in `ps_oplog` may be newer than the one in `ps_data__
`. Only when a full checkpoint has been downloaded, will the data be copied over to the individual tables. If multiple rows with the same table and id has been synced, only one will be preserved (the one with the highest `op_id`). \ No newline at end of file +The copy in `ps_oplog` may be newer than the one in `ps_data__
`. Only when a full checkpoint has been downloaded, will the data be copied over to the individual tables. If multiple rows with the same table and id has been synced, only one will be preserved (the one with the highest `op_id`). + + + If you run into limitations with the above JSON-based SQLite view system, check out [this experimental feature](/usage/use-case-examples/raw-tables) which allows you to define and manage raw SQLite tables to work around some limitations. We are actively seeking feedback about this functionality. + \ No newline at end of file From 724d8b0ee97342499af2cbf7946fd87c41ef0317 Mon Sep 17 00:00:00 2001 From: benitav Date: Fri, 11 Jul 2025 13:23:28 +0200 Subject: [PATCH 09/15] more fitting icon --- usage/use-case-examples.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/usage/use-case-examples.mdx b/usage/use-case-examples.mdx index c5e66958..dca3adbc 100644 --- a/usage/use-case-examples.mdx +++ b/usage/use-case-examples.mdx @@ -17,7 +17,7 @@ The following examples are available to help you get started with specific use c - + ## Additional Resources From 73eb7b76b0d1dd5d890124d24fd3be237f91ffbd Mon Sep 17 00:00:00 2001 From: benitav Date: Fri, 11 Jul 2025 13:24:24 +0200 Subject: [PATCH 10/15] Consolidate message --- usage/use-case-examples/raw-tables.mdx | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-) diff --git a/usage/use-case-examples/raw-tables.mdx b/usage/use-case-examples/raw-tables.mdx index 471a89be..20e89d0a 100644 --- a/usage/use-case-examples/raw-tables.mdx +++ b/usage/use-case-examples/raw-tables.mdx @@ -5,7 +5,12 @@ sidebarTitle: "Raw SQLite Tables" --- -Raw tables are an experimental feature for the purpose of getting feedback from users and refining the APIs. +Raw tables are an experimental feature. We're actively seeking feedback on: + +- API design and developer experience +- Additional features or optimizations needed + +Join our [Discord community](https://discord.gg/powersync) to share your experience and get help. By default, PowerSync uses a [JSON-based view system](/architecture/client-architecture#schema). Raw tables allow you to define native SQLite tables in the client-side schema, bypassing this. @@ -241,12 +246,3 @@ CREATE TRIGGER todo_lists_delete ## Migrations In PowerSync's [JSON-based view system](/architecture/client-architecture#schema) the client-side schema is applied to the schemaless data, meaning no migrations are required. Raw tables however are excluded from this, so it is the developers responsibility to manage migrations for these tables. - -## Support and Feedback - -Raw tables are an experimental feature. We're actively seeking feedback on: - -- API design and developer experience -- Additional features or optimizations needed - -Join our [Discord community](https://discord.gg/powersync) to share your experience and get help. \ No newline at end of file From d2eec19b77565300908deb7d28337d6f9dfe8ebc Mon Sep 17 00:00:00 2001 From: benitav Date: Fri, 11 Jul 2025 13:28:30 +0200 Subject: [PATCH 11/15] additional polish --- usage/use-case-examples/raw-tables.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/usage/use-case-examples/raw-tables.mdx b/usage/use-case-examples/raw-tables.mdx index 20e89d0a..e521b9cb 100644 --- a/usage/use-case-examples/raw-tables.mdx +++ b/usage/use-case-examples/raw-tables.mdx @@ -13,12 +13,12 @@ Raw tables are an experimental feature. We're actively seeking feedback on: Join our [Discord community](https://discord.gg/powersync) to share your experience and get help. -By default, PowerSync uses a [JSON-based view system](/architecture/client-architecture#schema). Raw tables allow you to define native SQLite tables in the client-side schema, bypassing this. +By default, PowerSync uses a [JSON-based view system](/architecture/client-architecture#schema) where data is stored schemalessly in JSON format and then presented through SQLite views based on the client-side schema. Raw tables allow you to define native SQLite tables in the client-side schema, bypassing this. This eliminates overhead associated with extracting values from the JSON data and provides access to advanced SQLite features like foreign key constraints and custom indexes. -*Availability** +**Availability** Raw tables were introduced in the following versions of our client SDKs: - TODO From c0471c7f40ca42b4e373d9a56eaf0e0a701f191d Mon Sep 17 00:00:00 2001 From: Simon Binder Date: Mon, 14 Jul 2025 17:37:04 +0200 Subject: [PATCH 12/15] Mention release versions for JS --- usage/use-case-examples/raw-tables.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/usage/use-case-examples/raw-tables.mdx b/usage/use-case-examples/raw-tables.mdx index e521b9cb..06f36dc3 100644 --- a/usage/use-case-examples/raw-tables.mdx +++ b/usage/use-case-examples/raw-tables.mdx @@ -21,7 +21,7 @@ This eliminates overhead associated with extracting values from the JSON data an **Availability** Raw tables were introduced in the following versions of our client SDKs: -- TODO +- __JavaScript__ (Node: `0.8.0`, React-Native: `1.23.0`, Web: `1.24.0`) Also note that raw tables are only supported by the new [Rust-based sync client](https://releases.powersync.com/announcements/improved-sync-performance-in-our-client-sdks), which is currently opt-in. From 008f9ef51cb34800a4c92f23e7f77011cd487bf2 Mon Sep 17 00:00:00 2001 From: Simon Binder Date: Tue, 15 Jul 2025 11:20:11 +0200 Subject: [PATCH 13/15] More notes on migrations --- usage/use-case-examples/raw-tables.mdx | 40 ++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) diff --git a/usage/use-case-examples/raw-tables.mdx b/usage/use-case-examples/raw-tables.mdx index 06f36dc3..0e787f93 100644 --- a/usage/use-case-examples/raw-tables.mdx +++ b/usage/use-case-examples/raw-tables.mdx @@ -246,3 +246,43 @@ CREATE TRIGGER todo_lists_delete ## Migrations In PowerSync's [JSON-based view system](/architecture/client-architecture#schema) the client-side schema is applied to the schemaless data, meaning no migrations are required. Raw tables however are excluded from this, so it is the developers responsibility to manage migrations for these tables. + +### Adding raw tables as a new table + +When you're adding new tables to your sync rules, clients will start to sync data on those tables - even if the tables aren't mentioned +in the client's schema yet. +So at the time you're introducing a new raw table to your app, it's possible that PowerSync has already synced some data for that +table, which would be stored in `ps_untyped`. When adding regular tables, PowerSync will automatically extract rows from `ps_untyped`. +With raw tables, that step is your responsibility. To copy data, run these statements in a transaction after creating the table: + +``` +INSERT INTO my_table (id, my_column, ...) + SELECT id, data ->> 'my_column' FROM ps_untyped WHERE type = 'my_table'; +DELETE FROM ps_untyped WHERE type = 'my_table'; +``` + +This does not apply if you've been using the raw table from the beginning (and never called `connect()` without them) - you only +need this for raw tables you already had locally. + +Another workaround is to clear PowerSync data when changing raw tables and opt for a full resync. + +### Migrating to raw tables + +To migrate from PowerSync-managed tables to raw tables, first: + +1. Open the database with the new schema mentioning raw tables. PowerSync will copy data from tables previously managed by PowerSync into + `ps_untyped`. +2. Create raw tables. +3. Run the `INSERT FROM SELECT` statement to insert `ps_untyped` data into your raw tables. + +### Migrations on raw tables + +When adding new columns to raw tables, there currently isn't a way to re-sync that table to add those columns from the server - we are +investigating possible workarounds and encourage users to try out if they need this. + +To ensure the column values are accurate, you'd have to delete all data after a migration and wait for the next complete sync. + +## Deleting data and raw tables + +APIs that clear an entire PowerSync database, like e.g. `disconnectAndClear()`, don't affect raw tables. +This should be kept in mind when you're using those methods - data from raw tables needs to be deleted explicitly. From e6b65eb125e519615758087ef88d3a083102c3ff Mon Sep 17 00:00:00 2001 From: Simon Binder Date: Wed, 16 Jul 2025 17:17:51 +0200 Subject: [PATCH 14/15] Add minimum versions for Dart and Kotlin --- usage/use-case-examples/raw-tables.mdx | 2 ++ 1 file changed, 2 insertions(+) diff --git a/usage/use-case-examples/raw-tables.mdx b/usage/use-case-examples/raw-tables.mdx index 0e787f93..ec4514c4 100644 --- a/usage/use-case-examples/raw-tables.mdx +++ b/usage/use-case-examples/raw-tables.mdx @@ -22,6 +22,8 @@ This eliminates overhead associated with extracting values from the JSON data an Raw tables were introduced in the following versions of our client SDKs: - __JavaScript__ (Node: `0.8.0`, React-Native: `1.23.0`, Web: `1.24.0`) +- __Dart__: Version 1.15.0 of `package:powersync`. +- __Kotlin__: Version 1.3.0 Also note that raw tables are only supported by the new [Rust-based sync client](https://releases.powersync.com/announcements/improved-sync-performance-in-our-client-sdks), which is currently opt-in. From 90fbe2f0b1076ecc012d06229a9ae4310cc33363 Mon Sep 17 00:00:00 2001 From: Simon Binder Date: Thu, 17 Jul 2025 09:14:02 +0200 Subject: [PATCH 15/15] Title case --- usage/use-case-examples/raw-tables.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/usage/use-case-examples/raw-tables.mdx b/usage/use-case-examples/raw-tables.mdx index ec4514c4..8791d4be 100644 --- a/usage/use-case-examples/raw-tables.mdx +++ b/usage/use-case-examples/raw-tables.mdx @@ -1,5 +1,5 @@ --- -title: "Raw SQLite Tables to bypass JSON View Limitations" +title: "Raw SQLite Tables to Bypass JSON View Limitations" description: "Use raw tables for native SQLite functionality and improved performance." sidebarTitle: "Raw SQLite Tables" ---