Manual compaction for segments

[hicder]

Problem

Right now, if a collection have multiple segments, we will go through each segment and do a search on it. Link. That means, if we have a lot of segments, query latency will suffer.

This task merges the segments if we have too many.

Detail

This will go and merge segments based on some criteria

• We will first inquire the collection to get a list of segments
• Users can then choose which segments to merge
• In the future, we will extend this to having a background thread to do this automatically, but for now, let's just focus on having users doing the manual compaction.

This task will contain:

• Have a gRPC to list what segments a collection has
• Have a gRPC to send a compact command with a list of segments
• These gRPC will have a corresponding API in the Collection struct
• Implement the logic to merge 2 segments (you'll need to familiarize yourself with builders and writers)
• Implement the logic to swap segments atomically

You'll need to understand the locking mechanism to make sure it doesn't have data race.