Backend: The chain indexer has no gap detection, reorg handling or backfill, so donations can be missed permanently

[Emmy123222]

Goal this serves — “Every transaction tracked on-chain” (README) — the indexer is what makes that true

Why this matters

The indexer is the component that makes the platform's central claim true: it watches project wallets and records donations that arrive on-chain, including ones no client ever reported. If it misses a payment, that donation exists on Stellar and does not exist on the platform — the donor's contribution is invisible, their leaderboard position is wrong, and the project's total understates reality.

It persists a cursor and resumes from it, which handles a clean restart. It does not handle the cases that actually occur.

Evidence

backend/src/services/indexerService.js maintains horizon_operations_cursor and flushes it periodically. A search across that file for reorg, gap, backfill or catch-up handling returns nothing.

Concretely: the cursor is flushed on an interval rather than per processed operation, so a crash between flushes reprocesses or skips depending on ordering; a project wallet added while the indexer is running is picked up only from that moment, with no backfill of prior donations; and there is no verification that the sequence of processed operations is contiguous.

Why this is hard

Detecting a gap requires knowing what should have been there. A cursor tells you where you are, not whether you skipped something. Contiguity has to be verified against the ledger rather than assumed.

Ledger history is large. Backfilling a newly added wallet from genesis is impractical; scoping the backfill window correctly, and being honest about what falls outside it, is a design decision.

Exactly-once against an at-least-once source. Reprocessing must be harmless, which means idempotency at the ingestion boundary rather than hoping the cursor is accurate.

Horizon is not always available. Extended downtime, rate limiting and pagination limits all need handling that does not silently drop a window of history.

Silence is ambiguous. No events can mean no donations or a broken indexer, and the system currently cannot tell those apart.

Suggested approach

Verify contiguity explicitly rather than trusting the cursor, and reconcile periodically against the chain to detect anything missed.

Make ingestion idempotent so reprocessing is safe, then treat the cursor as an optimisation rather than a correctness mechanism.

Support bounded backfill for newly added wallets and after extended downtime, and make indexer health observable so silence is distinguishable from failure.

Acceptance criteria

• Ingestion is idempotent: reprocessing an operation cannot create a duplicate donation, proven by a test.
• Gaps in processed history are detected rather than assumed absent, and detection is tested against a synthetic gap.
• A newly added project wallet is backfilled over a bounded, documented window.
• Extended Horizon downtime is recovered from without dropping a window of history.
• Rate limiting and pagination limits are handled with backoff rather than silent truncation.
• Indexer lag and health are observable, so a stalled indexer is distinguishable from a quiet period.
• A reconciliation pass compares on-chain payments to recorded donations and reports discrepancies.
• Cursor persistence semantics are documented, including exactly what a crash between flushes can cause.

Scope

Roughly 5,000–7,000 lines, including tests.

Relevant files

• backend/src/services/indexerService.js
• backend/src/services/sorobanEventIndexer.js
• backend/src/db/schema.sql
• docs/indexer.md