Feature: Add Snapshot functionality to Rust SDK and CLI #119
Conversation
Implements consistent point-in-time snapshot API in the rust SDK - discovers and creates tables and indexes from sqlite_schema - clones data for each table - fsync is switched off for faster writes Signed-off-by: Prateek Singh Rathore <[email protected]>
|
Hey @psrvere, interesting approach. The idea here is that the immediate transaction ensures that there are no new WAL entries, which seems to be safe. However, I suspect this is going to be pretty slow for larger databases because it's all going through the SQL interface as far as I can tell. IMHO, a better long-term strategy is to do this in turso.git and just make a copy of the database file itself while blocking concurrent writers. |
|
Thanks for the feedback @penberg. Yes, copying the file would be much faster. I saw a similar SQL based implementation in turso CLI and hence assumed that if its good there, it should be good in AgentFS. Let me do a file copy snapshot implementation in turso and share a draft PR soon. |
This addresses feedback from tursodatabase/agentfs#119 suggesting that snapshot functionality should be implemented in turso.git using direct fily copying rather than SQL-based row by row copying. Comment: https:://github.com/tursodatabase/agentfs/pull/119#issuecomment-3681336678 - extract core logic from Pager::checkpoint function to a new Pager::checkpoing_internal function and add a flag to keep_lock during the Finalize phase - create wrapper functions Pager::checkpoint, Pager::checkpoint_with_lock and Pager::block_checkpoint_keep_lock - add Connection::snapshot API that checkpoints while keeping the lock and copies the database file The lock is held to avoid a race condition after finishing checkpointing and before copying the file when concurrent writers can write to db. Signed-off-by: Prateek Singh Rathore <[email protected]>
This addresses feedback from tursodatabase/agentfs#119 suggesting that snapshot functionality should be implemented in turso.git using direct fily copying rather than SQL-based row by row copying. Comment: https:://github.com/tursodatabase/agentfs/pull/119#issuecomment-3681336678 - extract core logic from Pager::checkpoint function to a new Pager::checkpoing_internal function and add a flag to keep_lock during the Finalize phase - create wrapper functions Pager::checkpoint, Pager::checkpoint_with_lock and Pager::block_checkpoint_keep_lock - add Connection::snapshot API that checkpoints while keeping the lock and copies the database file The lock is held to avoid a race condition after finishing checkpointing and before copying the file when concurrent writers can write to db. Signed-off-by: Prateek Singh Rathore <[email protected]>
|
@penberg - I created tursodatabase/turso#4346 PR for adding snapshot API to Connection. Please review. |
2 participants
Closes #57
Closes #61
Implements consistent point-in-time snapshot API in the rust SDK