feat: Redis-backed payment cursor with distributed lock and Horizon rate-limit backoff#21
Conversation
Implements issue OrbitStream#7 — durable cursor persistence, leader election, and Horizon rate-limit backoff so the payment detector survives restarts and multi-instance deploys without gaps or double-processing. - RedisService / RedisModule: global ioredis connection with retry - PaymentCursorService: CAS cursor flush every 10 ops, 50-entry checkpoint log for rollback, SET NX EX distributed lock - PaymentDetectorService: rewired to inject PaymentCursorService; 429/5xx backoff with separate duration and interval knobs; rateLimitRemaining < 10 triggers the same 429 backoff path - app.module.ts: imports RedisModule (global, available everywhere) - payments.module.ts: registers PaymentCursorService as provider - stellar.service.ts: adds getPaymentsPage() with rate-limit headers and getHttpStatusFromError() for AxiosError unwrapping - package.json: adds ioredis (dep) and ioredis-mock (devDep) - .env.example: documents REDIS_URL and PLATFORM_RECEIVING_ACCOUNT - payment-cursor.spec.ts: 18 unit tests via ioredis-mock covering cursor store/restore, CAS race, lock acquire/contend/release/renew, checkpoint replay, rollback, and isAhead BigInt precision
|
Hi @nonsobethel0-dev! The implementation looks solid — nice work on the CAS cursor, checkpoint rollback, and distributed lock. CI is failing because Fix: Run this locally and commit the updated lock file: The |
|
Hi @nonsobethel0-dev! The implementation is solid — cursor persistence, distributed lock, Lua CAS, checkpoint rollback, and backoff logic all look correct and match the issue requirements. CI blocker: Missing test scenarios (the issue explicitly asks for these):
These are listed in the issue under "Testing" requirements. The current 18 unit tests cover the service methods well, but the issue specifically calls out these scenarios. |
|
Fixed the CI failure — |
…ix Redis test isolation - Add 6 crash recovery and full persist-and-restore integration tests to payment-cursor.spec.ts covering partial-batch crashes, unexpected cursor-ahead rollback, and CAS stale-replica rejection - Add payment-detector-backoff.spec.ts: 10 white-box tests covering 429 rate-limit backoff (10s/60s window), 5xx backoff (5s/30s window), and normal 3s default behaviour - Fix ioredis-mock test isolation: replace per-test new RedisMock() instances with a single sharedMock + flushall() in beforeEach to prevent data bleed between tests (fixes 2 pre-existing failures) - Apply Prettier formatting to payment-cursor.service.ts, payment-detector.service.ts, and stellar.service.ts
oomokaro1
left a comment
There was a problem hiding this comment.
LGTM. All previous feedback addressed — lock file included, crash recovery and backoff tests added, integration tests cover persist-and-restore cycles. One minor note: @types/ioredis is unnecessary since ioredis v5 ships its own types, but it won't break anything.
Closes #7
What changed
src/redis/) — global ioredis connection with exponential reconnect strategy and graceful shutdownsrc/payments/payment-cursor.service.ts) — durable cursor persistence using a Lua CAS script (no double-flush races), a 50-entry checkpoint log for safe rollback on restart, and a SET-NX-EX distributed lock so only one pod polls Horizon at a timePaymentCursorService; cursor flushed to Redis everyPERSIST_EVERY=10operations; distributed lock renewed each poll cycle; 429 and 5xx HTTP errors trigger separate backoff intervals and durations;rateLimitRemaining < 10triggers the 429 backoff path preemptivelygetPaymentsPage()returning Horizon rate-limit headers alongside records, andgetHttpStatusFromError()for clean AxiosError unwrappingPaymentCursorServiceas a providerRedisModule(marked@Global(), available app-wide)ioredis@^5.3.2(runtime) andioredis-mock@^8.9.0(test)REDIS_URLandPLATFORM_RECEIVING_ACCOUNTWhy
The old detector kept the cursor in memory only — a pod restart or crash silently skipped all payments processed since the last in-memory cursor update. Under horizontal scaling, multiple pods also raced against the same Horizon account stream. This change makes the cursor durable and the poller single-leader.
How to test
docker run -p 6379:6379 redis:alpine)REDIS_URL=redis://localhost:6379andPLATFORM_RECEIVING_ACCOUNT=<testnet address>in.envnpm installthennpm run dev— look forRedis connectedandRestored cursorlog linesnownpm test -- payment-cursor.spec— all 18 cases should pass via ioredis-mock