Skip to content

feat: Redis-backed payment cursor with distributed lock and Horizon rate-limit backoff#21

Merged
oomokaro1 merged 3 commits into
OrbitStream:mainfrom
nonsobethel0-dev:feat/issue-7-redis-cursor
Jun 18, 2026
Merged

feat: Redis-backed payment cursor with distributed lock and Horizon rate-limit backoff#21
oomokaro1 merged 3 commits into
OrbitStream:mainfrom
nonsobethel0-dev:feat/issue-7-redis-cursor

Conversation

@nonsobethel0-dev

@nonsobethel0-dev nonsobethel0-dev commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

Closes #7

What changed

  • RedisService / RedisModule (src/redis/) — global ioredis connection with exponential reconnect strategy and graceful shutdown
  • PaymentCursorService (src/payments/payment-cursor.service.ts) — durable cursor persistence using a Lua CAS script (no double-flush races), a 50-entry checkpoint log for safe rollback on restart, and a SET-NX-EX distributed lock so only one pod polls Horizon at a time
  • PaymentDetectorService — injected PaymentCursorService; cursor flushed to Redis every PERSIST_EVERY=10 operations; distributed lock renewed each poll cycle; 429 and 5xx HTTP errors trigger separate backoff intervals and durations; rateLimitRemaining < 10 triggers the 429 backoff path preemptively
  • StellarService — added getPaymentsPage() returning Horizon rate-limit headers alongside records, and getHttpStatusFromError() for clean AxiosError unwrapping
  • payments.module.ts — registers PaymentCursorService as a provider
  • app.module.ts — imports RedisModule (marked @Global(), available app-wide)
  • package.json — adds ioredis@^5.3.2 (runtime) and ioredis-mock@^8.9.0 (test)
  • .env.example — documents REDIS_URL and PLATFORM_RECEIVING_ACCOUNT

Why

The old detector kept the cursor in memory only — a pod restart or crash silently skipped all payments processed since the last in-memory cursor update. Under horizontal scaling, multiple pods also raced against the same Horizon account stream. This change makes the cursor durable and the poller single-leader.

How to test

  • Start Redis locally (docker run -p 6379:6379 redis:alpine)
  • Set REDIS_URL=redis://localhost:6379 and PLATFORM_RECEIVING_ACCOUNT=<testnet address> in .env
  • Run npm install then npm run dev — look for Redis connected and Restored cursor log lines
  • Kill and restart the server — confirm the cursor resumes from the last persisted token, not from now
  • Run the unit tests: npm test -- payment-cursor.spec — all 18 cases should pass via ioredis-mock

Implements issue OrbitStream#7 — durable cursor persistence, leader election, and
Horizon rate-limit backoff so the payment detector survives restarts and
multi-instance deploys without gaps or double-processing.

- RedisService / RedisModule: global ioredis connection with retry
- PaymentCursorService: CAS cursor flush every 10 ops, 50-entry
  checkpoint log for rollback, SET NX EX distributed lock
- PaymentDetectorService: rewired to inject PaymentCursorService;
  429/5xx backoff with separate duration and interval knobs;
  rateLimitRemaining < 10 triggers the same 429 backoff path
- app.module.ts: imports RedisModule (global, available everywhere)
- payments.module.ts: registers PaymentCursorService as provider
- stellar.service.ts: adds getPaymentsPage() with rate-limit headers
  and getHttpStatusFromError() for AxiosError unwrapping
- package.json: adds ioredis (dep) and ioredis-mock (devDep)
- .env.example: documents REDIS_URL and PLATFORM_RECEIVING_ACCOUNT
- payment-cursor.spec.ts: 18 unit tests via ioredis-mock covering
  cursor store/restore, CAS race, lock acquire/contend/release/renew,
  checkpoint replay, rollback, and isAhead BigInt precision
@oomokaro1

Copy link
Copy Markdown
Contributor

Hi @nonsobethel0-dev! The implementation looks solid — nice work on the CAS cursor, checkpoint rollback, and distributed lock. CI is failing because npm ci can't find the new dependencies in the lock file.

Fix: Run this locally and commit the updated lock file:

npm install
git add package-lock.json
git commit -m "chore: update package-lock.json for ioredis dependencies"
git push

The package.json adds ioredis, ioredis-mock, and their types, but package-lock.json wasn't included in the commit. npm ci requires them to be in sync.

@oomokaro1

Copy link
Copy Markdown
Contributor

Hi @nonsobethel0-dev! The implementation is solid — cursor persistence, distributed lock, Lua CAS, checkpoint rollback, and backoff logic all look correct and match the issue requirements.

CI blocker: package-lock.json is out of sync. Fix:

npm install
git add package-lock.json
git commit -m "chore: update package-lock.json for ioredis dependencies"
git push

Missing test scenarios (the issue explicitly asks for these):

  1. Rate limit handling test — simulate a 429 response from Horizon and verify the backoff kicks in (poll interval changes to 10s, resets after 1 minute)

  2. Crash recovery simulation — simulate a crash mid-processing (e.g. process 3 ops, then stop), restart, and verify the cursor rolls back to the last checkpoint

  3. Integration test — a full flow: store cursor → process payments → persist → restart → verify cursor restores correctly from Redis

These are listed in the issue under "Testing" requirements. The current 18 unit tests cover the service methods well, but the issue specifically calls out these scenarios.

@nonsobethel0-dev

Copy link
Copy Markdown
Contributor Author

Fixed the CI failure — npm ci was erroring because ioredis, ioredis-mock, and their transitive deps were added to package.json but package-lock.json was never updated alongside them. Regenerated the lock file and pushed. CI should be green now.

…ix Redis test isolation

- Add 6 crash recovery and full persist-and-restore integration tests
  to payment-cursor.spec.ts covering partial-batch crashes, unexpected
  cursor-ahead rollback, and CAS stale-replica rejection
- Add payment-detector-backoff.spec.ts: 10 white-box tests covering 429
  rate-limit backoff (10s/60s window), 5xx backoff (5s/30s window), and
  normal 3s default behaviour
- Fix ioredis-mock test isolation: replace per-test new RedisMock()
  instances with a single sharedMock + flushall() in beforeEach to
  prevent data bleed between tests (fixes 2 pre-existing failures)
- Apply Prettier formatting to payment-cursor.service.ts,
  payment-detector.service.ts, and stellar.service.ts

@oomokaro1 oomokaro1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. All previous feedback addressed — lock file included, crash recovery and backoff tests added, integration tests cover persist-and-restore cycles. One minor note: @types/ioredis is unnecessary since ioredis v5 ships its own types, but it won't break anything.

@oomokaro1 oomokaro1 merged commit 18dc866 into OrbitStream:main Jun 18, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEAT] Redis-Backed Payment Detection Cursor with Failover

2 participants