Start offset from maximum number instead of zero

[frisbeeman]

Why read all the files if we can do file naming with reverse sorting and count only the last file (the LIST query sorts files by ASC)?

Originally posted by @s-prosvirnin in #24 (comment)

[coderabbitai]

📝 CodeRabbit Plan Mode

Generate an implementation plan and prompts that you can use with your favorite coding agent.

✅ Successfully generated implementation plan.

💡 Use @coderabbitai <feedback> to replan

Examples

• Example 1
• Example 2

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[coderabbitai]

Coding Plan

Approach

Optimize the queue recovery process by using PaginatedListStore to support ordered listing, potentially enabling us to find the maximum offset per topic more efficiently without processing all segment files.

Observations

The queue system stores segments as Parquet files in an object_store-backed storage with a hierarchical path structure {topic}/{offset}.parquet. Files use 20-digit zero-padded offsets enabling lexicographic sorting. The recovery process (crates/icegate-queue/src/writer.rs) currently scans all segments during startup to determine the maximum offset per topic using an O(n) max-tracking approach with a HashMap.

Assumptions

Assumption 1: Pagination Strategy for Maximum Offset Discovery

Options Considered:

• Enumerate all segments with pagination and track maximum (similar to current approach but with pagination)
• Use per-topic pagination with list_with_delimiter to list each topic separately and find its maximum
• Attempt to leverage lexicographic ordering within pages to early-exit when maximum is found

Chosen Option: Use per-topic pagination with list_with_delimiter

Rationale: This approach lists topics first using list_with_delimiter to get common_prefixes, then paginates through each topic's segments separately. This provides better topic isolation, reduces memory usage by processing one topic at a time, and aligns with the hierarchical path structure. While we still need to process all segments per topic (since pagination doesn't guarantee sorted results globally), this approach is more memory-efficient and sets up the foundation for future optimizations if backends provide ordered results.

💡 User Tips

Regenerate the plan with different choices with @coderabbitai <feedback>.

Implementation Steps

Phase 1: Integrate PaginatedListStore

This phase adds PaginatedListStore support to the QueueWriter for paginated segment enumeration during recovery.

Task 1: Add PaginatedListStore Wrapper

Extend the QueueWriter in crates/icegate-queue/src/writer.rs to:

• Add support for using PaginatedListStore alongside the existing ObjectStore
• Create a helper method that wraps the store with PaginatedListStore capabilities
• Ensure the wrapper is initialized during QueueWriter construction
• Maintain backward compatibility with existing ObjectStore usage in write operations

Task 2: Implement Paginated Topic Discovery

Add a new method in crates/icegate-queue/src/writer.rs for discovering topics using pagination:

• Use list_with_delimiter on the base path to enumerate topic prefixes (common_prefixes)
• Return a list of discovered topic paths for further processing
• Handle empty stores gracefully (return empty list when no topics exist)
• Add error handling for pagination failures

🤖 Prompt for AI agents

Integrate PaginatedListStore into QueueWriter to enable paginated segment
enumeration during recovery.

In `crates/icegate-queue/src/writer.rs`:
- Add PaginatedListStore support to QueueWriter alongside existing ObjectStore
- Create helper method to wrap store with PaginatedListStore capabilities
- Initialize wrapper during QueueWriter construction
- Maintain backward compatibility for existing ObjectStore write operations
- Implement new method for discovering topics using pagination with
list_with_delimiter
- Return discovered topic paths from common_prefixes
- Handle empty stores gracefully by returning empty list
- Add error handling for pagination failures

Phase 2: Optimize Recovery with Pagination

This phase modifies the recovery process to use pagination for more efficient segment enumeration.

Task 1: Implement Per-Topic Paginated Recovery

Modify the recover() method in crates/icegate-queue/src/writer.rs to:

• Call the topic discovery method from Task 1.2 to get list of topics
• For each discovered topic, use PaginatedListStore to enumerate segments with pagination
• Process each page of results, tracking the maximum offset for the current topic
• Update the topic's offset after processing all its pages
• Maintain the existing offset storage mechanism (set_offset calls)

Task 2: Optimize Memory Usage During Recovery

Enhance the paginated recovery implementation in crates/icegate-queue/src/writer.rs:

• Process segment pages one at a time rather than collecting all items into memory
• Use streaming iteration over paginated results to reduce memory footprint
• Track only the maximum offset per topic (maintain O(t) space complexity where t is topic count)
• Add configuration for page size to balance network requests vs memory usage

🤖 Prompt for AI agents

Modify recovery process to use pagination for efficient segment enumeration.

In `crates/icegate-queue/src/writer.rs`:
- Update `recover()` method to call topic discovery method and get list of
topics
- For each topic, use PaginatedListStore to enumerate segments with pagination
- Process each page of results and track maximum offset for the current topic
- Update topic offset after processing all pages using existing set_offset
mechanism
- Process segment pages one at a time using streaming iteration instead of
collecting into memory
- Track only maximum offset per topic to maintain O(t) space complexity
- Add configuration for page size to balance network requests vs memory usage

Phase 3: Add Tests and Documentation

This phase ensures the pagination-based recovery is well-tested and documented.

Task 1: Add Comprehensive Recovery Tests

Create test cases in crates/icegate-queue/src/writer.rs:

• Test paginated recovery with empty store
• Test paginated recovery with single topic and multiple segments
• Test paginated recovery with multiple topics
• Test paginated recovery with nested topic paths (e.g., tenant/acme/logs)
• Test recovery behavior with different page sizes
• Verify that paginated recovery produces same results as original implementation

Task 2: Document Pagination Behavior and Benefits

Add documentation to crates/icegate-queue/src/writer.rs:

• Document the pagination-based recovery approach in module-level comments
• Explain memory usage benefits of per-topic pagination
• Note that while pagination is used, we still process all segments (not relying on assumed ordering)
• Document the page size configuration and its trade-offs
• Add comments in the recover() method explaining the per-topic enumeration flow

🤖 Prompt for AI agents

Add comprehensive tests and documentation for pagination-based recovery.

In `crates/icegate-queue/src/writer.rs`:
- Create test cases for paginated recovery with empty store
- Test single topic with multiple segments
- Test multiple topics
- Test nested topic paths like `tenant/acme/logs`
- Test different page sizes
- Verify paginated recovery produces same results as original implementation
- Add module-level documentation explaining pagination-based recovery approach
- Document memory usage benefits of per-topic pagination
- Note that all segments are still processed (not relying on assumed ordering)
- Document page size configuration and trade-offs
- Add inline comments in recover() method explaining per-topic enumeration flow

---

🚀 Next Steps

🤖 All AI agent prompts combined

Task: 1

Integrate PaginatedListStore into QueueWriter to enable paginated segment
enumeration during recovery.

In `crates/icegate-queue/src/writer.rs`:
- Add PaginatedListStore support to QueueWriter alongside existing ObjectStore
- Create helper method to wrap store with PaginatedListStore capabilities
- Initialize wrapper during QueueWriter construction
- Maintain backward compatibility for existing ObjectStore write operations
- Implement new method for discovering topics using pagination with
list_with_delimiter
- Return discovered topic paths from common_prefixes
- Handle empty stores gracefully by returning empty list
- Add error handling for pagination failures
===============================================================================

Task: 2

Modify recovery process to use pagination for efficient segment enumeration.

In `crates/icegate-queue/src/writer.rs`:
- Update `recover()` method to call topic discovery method and get list of
topics
- For each topic, use PaginatedListStore to enumerate segments with pagination
- Process each page of results and track maximum offset for the current topic
- Update topic offset after processing all pages using existing set_offset
mechanism
- Process segment pages one at a time using streaming iteration instead of
collecting into memory
- Track only maximum offset per topic to maintain O(t) space complexity
- Add configuration for page size to balance network requests vs memory usage
===============================================================================

Task: 3

Add comprehensive tests and documentation for pagination-based recovery.

In `crates/icegate-queue/src/writer.rs`:
- Create test cases for paginated recovery with empty store
- Test single topic with multiple segments
- Test multiple topics
- Test nested topic paths like `tenant/acme/logs`
- Test different page sizes
- Verify paginated recovery produces same results as original implementation
- Add module-level documentation explaining pagination-based recovery approach
- Document memory usage benefits of per-topic pagination
- Note that all segments are still processed (not relying on assumed ordering)
- Document page size configuration and trade-offs
- Add inline comments in recover() method explaining per-topic enumeration flow

💡 Iterate on the plan with: @coderabbitai <feedback>

Example Feedback
- @coderabbitai You can skip phase 3. Add a simple unit test case for phase 2.
- @coderabbitai For assumption 1 go ahead with option 3 and replan.

---

💬 Have feedback or questions? Drop into our discord!

[frisbeeman]

@coderabbitai Use PaginatedListStore to support ordered listing

[coderabbitai]

I've updated the plan above based on your feedback. Ready to proceed when you are!