-
Notifications
You must be signed in to change notification settings - Fork 21
Proposal: [MySQL] Add replication keep alive. #314
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
…kage directly Added check for tablemap events
…tener class. Introduced a mechanism to limit the maximum size of the binlog processing queue, thus also limiting memory usage. This maximum processing queue size is configurable
# Conflicts: # modules/module-mysql/package.json
Cleaned up BinLogStream logs a bit
Simplified BinLogListener stopping mechanism
Added a few more defensive stopped checks to the binlog listener
… memory usage rather than number of events
… memory usage rather than number of events. Introduced a maximum timeout that the binlog processing queue can be paused before auto-resuming. This is to prevent the replication connection timing out.
Made SourceTable implement SourceEntityDescriptor interface
…trics instead of ignoring them. SourceTable. Moved MySQL table detail retrieval logic to utility function.
# Conflicts: # modules/module-mongodb-storage/src/storage/implementation/MongoBucketBatch.ts # modules/module-mongodb-storage/src/storage/implementation/MongoSyncBucketStorage.ts # modules/module-mongodb/src/replication/ChangeStream.ts # modules/module-mysql/package.json # modules/module-mysql/src/replication/BinLogStream.ts # modules/module-mysql/src/replication/zongji/BinLogListener.ts # modules/module-mysql/src/replication/zongji/zongji-utils.ts # modules/module-mysql/test/src/BinLogListener.test.ts # modules/module-postgres-storage/src/storage/PostgresSyncRulesStorage.ts # modules/module-postgres/src/replication/WalStream.ts # packages/service-core/src/storage/SourceTable.ts # pnpm-lock.yaml
Improved binlog table filtering Added extended type definitions for node-sql-parser package
Cleaned up MySQL tests in general and added a few new test utils
… on the mysql database yet.
TableFilter creation is now internally handled in the BinLog listener Pause/unpause binlog listening now uses the same stop start functionality used for schema change handling.
…on-keepalive-mechanism
Added tests for keepalive Moved common test functions to utils
# Conflicts: # modules/module-mysql/src/replication/BinLogStream.ts # modules/module-mysql/test/src/BinLogListener.test.ts # modules/module-mysql/test/src/util.ts
|
When the source MySQL database is idle for long periods of time, write checkpoints in powersync will be sparse.
To get around this, we make regular updates to a powersync keepalive table on the source database. This then ensures regular commit events, and correspondingly regular write checkpoints.
If the PowerSync MySQL user has got the appropriate permissions, PowerSync can create and maintain the keepalive table.
If instead the MySQL user has more limited permissions, it would be up to the user to create the table themselves. The Powersync MySQL user would still need a minimal number of permissions on the table though for this to work:
If the table has not been pre-created and the PowerSync MySQL user does not have the required permissions, the keepalive mechanism is disabled and a warning is logged.