[FEAT] Redis-Backed Payment Detection Cursor with Failover
Priority: High
Difficulty: Hard
Estimated Effort: 3-4 days
Relevant Packages: OrbitStream_backend/
Labels: enhancement, infrastructure, priority:high
Requirements
1. Redis Cursor Storage
- Store the cursor in Redis using a dedicated key:
orbitstream:payment_cursor:{account}
- Use Redis
GET/SET for cursor persistence
- Persist cursor to Redis every 10 processed operations (not every single one — performance)
- On startup, restore cursor from Redis. If Redis has no cursor, start from "now" and log a warning
- Set a TTL on the cursor key (24 hours) so stale cursors from decommissioned accounts don't persist forever
2. Distributed Lock (Leader Election)
- Only one instance should poll payments at a time
- Implement a Redis-based distributed lock using
SET NX EX (set if not exists, with expiry)
- Lock key:
orbitstream:payment_lock:{account}
- Lock TTL: 30 seconds (longer than the 3-second polling interval)
- Before each poll cycle, renew the lock (extend TTL)
- If lock acquisition fails, the instance skips polling (doesn't poll, doesn't process)
- If an instance crashes, the lock expires automatically and another instance can take over
3. Atomic Cursor Update
- Use a Redis Lua script for atomic cursor read-check-update:
local current = redis.call('GET', KEYS[1])
if current == ARGV[1] or current == false then
redis.call('SET', KEYS[1], ARGV[2], 'EX', ARGV[3])
return 1
end
return 0
- This prevents two instances from updating the cursor to different values simultaneously
4. Checkpoint and Replay
- Maintain a checkpoint log in Redis:
orbitstream:payment_checkpoint:{account}
- Store the last 50 processed operation paging tokens
- On startup, if the cursor is ahead of the checkpoint, roll back to the last checkpoint
- This ensures at-least-once delivery: if the process crashes mid-processing, it replays from the last safe point
5. Horizon Rate Limit Handling
- Implement adaptive polling interval:
- Default: 3 seconds
- If Horizon returns 429: back off to 10 seconds for 1 minute
- If Horizon returns 5xx: back off to 5 seconds for 30 seconds
- If Horizon returns 200 with empty results: stay at 3 seconds
- Track Horizon rate limit headers (
X-RateLimit-Limit, X-RateLimit-Remaining) and slow down proactively when remaining < 10
6. Testing
- Unit tests with
ioredis-mock: cursor storage, lock acquisition, atomic update
- Test lock contention: two instances trying to acquire the same lock
- Test cursor restore: verify cursor is restored from Redis on startup
- Test checkpoint replay: simulate crash mid-processing, verify replay from checkpoint
- Test rate limit handling: simulate 429 responses, verify backoff
- Integration test: full payment flow with Redis-backed cursor
[FEAT] Redis-Backed Payment Detection Cursor with Failover
Priority: High
Difficulty: Hard
Estimated Effort: 3-4 days
Relevant Packages:
OrbitStream_backend/Labels:
enhancement,infrastructure,priority:highRequirements
1. Redis Cursor Storage
orbitstream:payment_cursor:{account}GET/SETfor cursor persistence2. Distributed Lock (Leader Election)
SET NX EX(set if not exists, with expiry)orbitstream:payment_lock:{account}3. Atomic Cursor Update
4. Checkpoint and Replay
orbitstream:payment_checkpoint:{account}5. Horizon Rate Limit Handling
X-RateLimit-Limit,X-RateLimit-Remaining) and slow down proactively when remaining < 106. Testing
ioredis-mock: cursor storage, lock acquisition, atomic update