Skip to content

feat: Implement SSTable V2 Reader with Bloom Filter and Block Caching#34

Merged
ElioNeto merged 20 commits into
developfrom
fix/sstable-reader-missing-features
Feb 4, 2026
Merged

feat: Implement SSTable V2 Reader with Bloom Filter and Block Caching#34
ElioNeto merged 20 commits into
developfrom
fix/sstable-reader-missing-features

Conversation

@ElioNeto

@ElioNeto ElioNeto commented Feb 4, 2026

Copy link
Copy Markdown
Owner

Summary

This PR implements a complete SSTable V2 reader with advanced features including Bloom filters, sparse indexing, and block-level caching.

Changes Made

1. SSTable Reader Implementation (src/storage/reader.rs)

  • Bloom Filter Integration: Fast key existence check before disk access
  • Sparse Index: Binary search on block-level index for efficient lookups
  • LRU Block Cache: Configurable block caching to reduce disk I/O
  • Compression Support: LZ4 decompression for blocks and metadata
  • Boundary Validation: Min/max key checks before binary search

2. Block Structure Updates (src/storage/block.rs)

  • Made data and offsets fields pub(crate) for access from reader module
  • Maintained encapsulation while allowing internal access

3. Key Features

Performance Optimizations

  • Bloom Filter: O(1) negative lookups with configurable false positive rate
  • Block Caching: LRU cache reduces repeated disk reads
  • Sparse Index: O(log n) block lookup using binary search
  • Lazy Loading: Only loads required blocks on demand

Correctness

  • Borrow Checker Compliance: Proper cloning to avoid lifetime issues
  • Type Safety: Correct Bloom filter deserialization
  • Boundary Checks: Validates key ranges before search
  • Data Integrity: Verifies decompressed block sizes

4. API Surface

pub struct SstableReader {
    // Public methods:
    pub fn open(path: PathBuf, config: StorageConfig) -> Result<Self>
    pub fn get(&mut self, key: &str) -> Result<Option<LogRecord>>
    pub fn might_contain(&self, key: &str) -> bool
    pub fn scan(&mut self) -> Result<Vec<(Vec<u8>, LogRecord)>>
    pub fn metadata(&self) -> &MetaBlock
    pub fn path(&self) -> &PathBuf
}

5. Test Coverage

Comprehensive test suite covering:

  • ✅ Basic roundtrip (write + read)
  • ✅ Bloom filter false positive rate
  • ✅ Multiple block handling
  • ✅ Boundary key cases (min/max)
  • ✅ Full table scan
  • ✅ Invalid file format detection

Technical Details

Bloom Filter

  • Uses bloomfilter crate v3.0.1
  • Serialized/deserialized using from_bytes/into_bytes
  • Configurable FP rate via StorageConfig

Block Iteration

  • Manual offset-based iteration over block data
  • Reads key/value lengths as u16 (little-endian)
  • Decodes LogRecord from value bytes

Cache Configuration

  • Cache capacity = block_cache_size_mb * 1024 * 1024 / block_size
  • Default: 100 blocks minimum
  • Keyed by block offset (u64)

Fixes

  • ✅ Resolved borrow checker errors by cloning BlockMeta
  • ✅ Fixed Bloom filter type mismatch (removed unnecessary reference)
  • ✅ Made Block fields accessible within storage module
  • ✅ Changed search_in_block to static method to avoid self borrow

Testing

cargo test --all-features
cargo build --release --all-features

All tests pass ✅

Related Issues

Closes #3 (if exists)

Checklist

  • Code compiles without errors
  • All tests pass
  • Added comprehensive test coverage
  • Documentation added for public APIs
  • Borrow checker compliance
  • No clippy warnings

@ElioNeto
ElioNeto merged commit 731b826 into develop Feb 4, 2026
2 checks passed
@github-actions github-actions Bot mentioned this pull request Feb 4, 2026
@ElioNeto
ElioNeto deleted the fix/sstable-reader-missing-features branch February 4, 2026 12:38
This was referenced Feb 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant