[platinumhamburg]

Purpose

Linked issue: closes #4769

Add asynchronous manual compaction for ordinary primary-key tables. Operators can optimize the LSM, reclaim eligible obsolete state, or migrate existing SSTs to the configured index layout. MIGRATE skips physical rewriting when live SSTs already match the cluster configuration.

Brief change log

• Add partition/table/database/cluster Admin overloads and compact, list_compactions, get_compaction, and cancel_compaction Flink procedures.
• Persist immutable fixed plans, successful bucket/snapshot deltas, and task summaries using remote metadata files and bounded ZooKeeper records. Keep one active operation per cluster and one bucket per TS, with no task queue.
• Require a new confirmed KV snapshot and a published completion record for every successful bucket, including no-op migration. Preserve known-ID snapshot reconciliation when a commit response is lost.
• Use idempotent bucket starts to dispatch and observe work without a separate bucket-status RPC. Copy requests and returned statuses, and reuse successful results only within the same leader epoch.
• Bound recovery by the published successful-bucket count and actual serialized payload limits; fence progress publication with Coordinator epoch and atomic task/active CAS.
• Use at-least-once scheduling: retain published successes on recovery and reschedule remaining buckets against current leaders. Fence stale execution with Coordinator/leader epochs, preserve the original absolute deadline, and retry transient network errors or occupied TS slots on subsequent control cycles.
• Keep native compaction on a dedicated worker, preserve resource ownership through actual exit, and defer KV retirement/deletion until native and snapshot users release their resources.
• Reuse existing disk/write-pressure admission guards, coalesce normal publication on the control cycle, and expose progress, cancellation, bounded errors, and retention cleanup. Failed operations count every remaining bucket as failed; cancelled operations count every remaining bucket as cancelled.

Tests

• Current-head focused RPC/server reactor: 145 tests passed, covering contiguous API keys, message codecs, compaction scheduling/persistence, RocksDB, snapshots, prefix lookup, and replica lifecycle.
• Flink 1.18 compaction procedures: 3 tests passed, including progress serialization through the procedure context.
• Spark lake suite using freshly installed, packaged dependencies: all 54 tests passed, including the time-range case that timed out in the earlier CI run.
• Client CompactionITCase: 2 tests passed, covering real SST migration, committed-snapshot recovery, repeated MIGRATE without rewriting, and OPTIMIZE/RECLAIM with foreground writes.
• Checkstyle and Spotless checks passed in the affected reactors. Rust protocol regeneration produced no additional changes.

Remote CI is still pending. The full repository suite and actual JDK 8 runtime/API compatibility are not claimed verified. The combined scenario of held native work with real three-TS leader/active-Coordinator failover remains untested; deterministic component tests cover those failure mechanisms separately.

API and Format

Additive Java Admin interfaces, Flink procedures, RPC messages/API keys, exceptions, and operation metadata. Existing SST and KV snapshot formats remain in use; rewritten SSTs follow the effective cluster configuration. No caller-supplied request ID or index type is introduced.

Documentation

Add an operations guide and Flink procedure reference/examples.