Complete compaction system: full Size-Tiered and Leveled strategies, tombstone GC, status API

[ElioNeto]

Problem

According to ROADMAP.md, compaction is listed as "not started" for v3-lts, but significant compaction infrastructure already exists:

• src/core/engine/compaction.rs — implements Compaction struct with pick_compaction() and compact() methods
• CompactionStrategy trait with SizeTieredCompaction and LeveledCompaction implementations
• benchmarks/write_amplification.rs — measures compaction write amplification
• POST /admin/compact — triggers compaction via API
• engine.maybe_compact() — background compaction spawning
• version_set.atomic_replace() — atomic SSTable replacement after compaction

However, there are gaps:

Missing features

1. Tombstone GC during compaction: compact_cf_core() passes range tombstones to the compaction strategy, but the Leveled compaction strategy may not fully remove tombstones when compacting levels beyond L0
2. Compaction status/progress API: No way to query current compaction progress, pending work, or estimated completion
3. Compaction metrics: CompactionMetrics exists but is only partially plumbed through the API (/admin/compact returns metrics, but background compaction metrics are lost)
4. Manual compaction targeting: Cannot compact a specific key range or a specific SSTable
5. Compaction cancellation: No way to cancel a running compaction
6. Compaction strategy configuration: Cannot change compaction strategy at runtime

What ROADMAP.md promises for v3-lts

• Full Size-Tiered and Leveled compaction
• Tombstone garbage collection
• Compaction status API
• Compaction metrics with Prometheus integration
• Compaction strategy selection per column family

Request

1. Audit current compaction:

   • List all existing compaction code paths
   • Identify which ROADMAP items are already done vs. truly missing
   • Update ROADMAP.md with accurate status

2. Complete missing compaction features:

   • Ensure tombstone GC works for all strategy types
   • Add GET /admin/compaction/status endpoint returning pending/paused/running state
   • Add compaction metrics to GET /stats output
   • Allow strategy selection per column family (via engine option or config)

3. Add compaction tests:

   • Tombstone removal after compaction
   • Cross-level compaction correctness
   • Compaction with concurrent writes
   • Compaction interruption + recovery

Acceptance criteria

• ROADMAP.md accurately reflects compaction implementation status
• All compaction strategies pass tombstone GC tests
• Compaction metrics are exposed via API
• Compaction status queryable via API
• No regression in existing tests