Skip to content

Proposal: [MySQL] Add replication keep alive. #314

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 67 commits into
base: main
Choose a base branch
from

Conversation

Rentacookie
Copy link
Contributor

@Rentacookie Rentacookie commented Jul 30, 2025

When the source MySQL database is idle for long periods of time, write checkpoints in powersync will be sparse.

To get around this, we make regular updates to a powersync keepalive table on the source database. This then ensures regular commit events, and correspondingly regular write checkpoints.

If the PowerSync MySQL user has got the appropriate permissions, PowerSync can create and maintain the keepalive table.

If instead the MySQL user has more limited permissions, it would be up to the user to create the table themselves. The Powersync MySQL user would still need a minimal number of permissions on the table though for this to work:

CREATE TABLE IF NOT EXISTS
   <database>.powersync_keepalive(
	id INT PRIMARY KEY DEFAULT 1,
	last_ping TIMESTAMP(3) DEFAULT CURRENT_TIMESTAMP(3)
)
GRANT SELECT, INSERT, UPDATE ON <database>.powersync_keepalive TO 'repl_user'@'%';

If the table has not been pre-created and the PowerSync MySQL user does not have the required permissions, the keepalive mechanism is disabled and a warning is logged.

…kage directly

Added check for tablemap events
…tener class.

Introduced a mechanism to limit the maximum size of the binlog processing queue, thus also limiting memory usage.
This maximum processing queue size is configurable
# Conflicts:
#	modules/module-mysql/package.json
Cleaned up BinLogStream logs a bit
Simplified BinLogListener stopping mechanism
Added a few more defensive stopped checks to the binlog listener
… memory usage rather than number of events.

Introduced a maximum timeout that the binlog processing queue can be paused before auto-resuming. This is to prevent the replication connection timing out.
Made SourceTable implement SourceEntityDescriptor interface
…trics instead of ignoring them. SourceTable.

Moved MySQL table detail retrieval logic to utility function.
# Conflicts:
#	modules/module-mongodb-storage/src/storage/implementation/MongoBucketBatch.ts
#	modules/module-mongodb-storage/src/storage/implementation/MongoSyncBucketStorage.ts
#	modules/module-mongodb/src/replication/ChangeStream.ts
#	modules/module-mysql/package.json
#	modules/module-mysql/src/replication/BinLogStream.ts
#	modules/module-mysql/src/replication/zongji/BinLogListener.ts
#	modules/module-mysql/src/replication/zongji/zongji-utils.ts
#	modules/module-mysql/test/src/BinLogListener.test.ts
#	modules/module-postgres-storage/src/storage/PostgresSyncRulesStorage.ts
#	modules/module-postgres/src/replication/WalStream.ts
#	packages/service-core/src/storage/SourceTable.ts
#	pnpm-lock.yaml
Improved binlog table filtering
Added extended type definitions for node-sql-parser package
Cleaned up MySQL tests in general and added a few new test utils
TableFilter creation is now internally handled in the BinLog listener
Pause/unpause binlog listening now uses the same stop start functionality used for schema change handling.
Added tests for keepalive
Moved common test functions to utils
# Conflicts:
#	modules/module-mysql/src/replication/BinLogStream.ts
#	modules/module-mysql/test/src/BinLogListener.test.ts
#	modules/module-mysql/test/src/util.ts
Copy link

changeset-bot bot commented Jul 30, 2025

⚠️ No Changeset found

Latest commit: 4f0315b

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@Rentacookie Rentacookie changed the title Mysql replication keepalive mechanism [MySQL] Add replication keep alive Jul 30, 2025
@Rentacookie Rentacookie requested a review from rkistner July 30, 2025 09:21
@Rentacookie Rentacookie changed the title [MySQL] Add replication keep alive Proposal: [MySQL] Add replication keep alive. Jul 30, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant