[MongoDB Storage] Compact parameter lookups

[rkistner]

The compact action now also compacts bucket parameter storage.

I many use cases, parameter lookups are not modified often, and take up fairly little storage. However, there are some cases where it is modified often, and this causes two issues:

1. Storage increases indefinitely.
2. Parameter queries gets slower.

This compacts the parameter storage when using MongoDB storage.

See the new docs/parameters-lookups.md for details on the implementation.

One notable change is that we now do "snapshot" queries for parameter data, matching the snapshot state of the checkpoint query. This is required for consistency of parameter lookups during a compact operation. This required some refactoring to the storage APIs.

TODO:

• Add tests specific to removing deleted parameter lookup entries.

[changeset-bot]

🦋 Changeset detected

Latest commit: d2bc507

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 11 packages

Name	Type
@powersync/service-module-postgres-storage	Minor
@powersync/service-module-mongodb-storage	Minor
@powersync/service-core-tests	Minor
@powersync/service-core	Minor
@powersync/service-image	Minor
@powersync/service-schema	Minor
@powersync/service-module-mongodb	Patch
@powersync/service-module-mysql	Patch
@powersync/service-module-postgres	Patch
@powersync/service-module-core	Patch
test-client	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[Copilot]

Pull Request Overview

This PR enhances the MongoDB storage implementation to include parameter lookup compaction alongside existing bucket data compaction. The compact action now also removes superseded parameter storage entries, addressing storage growth and query performance issues in use cases where parameter lookups are modified frequently.

Key changes:

• Added parameter data compaction with snapshot consistency guarantees
• Refactored storage APIs to use snapshot-based parameter queries instead of checkpoint-based queries
• Implemented comprehensive test coverage for parameter compaction scenarios

Reviewed Changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
packages/service-core/src/storage/SyncRulesBucketStorage.ts	Adds compactParameterData option and moves getParameterSets to ReplicationCheckpoint interface
packages/service-core/src/sync/BucketChecksumState.ts	Updates parameter lookup calls to use checkpoint's getParameterSets method
packages/service-core/src/entry/commands/compact-action.ts	Modifies compact command to enable parameter compaction when no specific buckets are targeted
packages/service-core/test/src/sync/BucketChecksumState.test.ts	Updates test mocks to use new checkpoint-based parameter lookup API
packages/service-core-tests/src/tests/register-parameter-compacting-tests.ts	Adds comprehensive tests for parameter compaction functionality
modules/module-postgres-storage/src/storage/PostgresSyncRulesStorage.ts	Implements new checkpoint-based parameter lookup interface for Postgres
modules/module-mongodb-storage/src/storage/implementation/MongoSyncBucketStorage.ts	Implements snapshot-based parameter queries and integrates parameter compaction
modules/module-mongodb-storage/src/storage/implementation/MongoParameterCompactor.ts	New compactor implementation for parameter lookup data

Comments suppressed due to low confidence (1)

packages/service-core-tests/src/tests/register-parameter-compacting-tests.ts:78

• The test does not verify that parameter compaction actually occurred by checking storage metrics before and after compaction. Consider adding assertions to validate that the parameter storage size decreased after compaction.

    await bucketStorage.compact({ compactParameterData: true });

[stevensJourney]

This is a very cool use-case for MongoDB snapshot queries. The changes here look good to me, I could not spot any potential issues :D