Backend: There is no safe way to rebuild a projection, so a bug in read-model logic is unrecoverable

[Emmy123222]

Goal this serves — Trustworthy totals — the event store is only useful if it can be replayed

Why this matters

The platform is event-sourced: event_stream is the authoritative record and read models are derived from it. The entire point of that design is that a derived view can be discarded and rebuilt from the events. There is no mechanism to do so.

This is not hypothetical. A double-counting bug inflated donation totals, and the fix could only stop new damage — the existing wrong values remain because nothing can recompute them. Every future projection bug has the same permanent consequence.

Evidence

backend/src/eventSourcing/ contains an event store, a command bus and projections, and schema.sql carries idx_event_stream_processed, so processing position is tracked. What does not exist is any way to reset that position, rebuild a projection from scratch, or run a new projection version alongside the old one.

Issue #368 records the concrete consequence: totals inflated by a historical bug, with recomputation from event_stream identified as the fix and nothing available to perform it.

Why this is hard

Rebuilding while serving traffic. Taking the site down to rebuild is not acceptable, so a rebuild has to run alongside live projection, catch up to the live position, and switch over atomically.

Event schema evolves. Events written a year ago may lack fields today's projection expects. Versioned events and upcasting are unavoidable once replay is real.

Side effects must not replay. Projections that send email or enqueue jobs cannot be naively re-run — rebuilding must not re-notify every donor in the platform's history.

Correctness needs proving. A rebuild that silently produces different numbers is worse than none. There must be a way to compare a rebuilt projection against the live one before switching.

Suggested approach

Make projections explicitly versioned and independently positioned so a new version can be built from the beginning of the stream while the current one keeps serving.

Separate pure derivation from side effects so replay is safe by construction rather than by remembering to disable notifications.

Provide a comparison mode that reports divergence between a rebuilt and live projection before any switchover, and make the switch atomic and reversible.

Use this to recompute the totals #368 describes as the first real exercise of the mechanism.

Acceptance criteria

• A projection can be rebuilt from the start of the event stream while the current one continues serving.
• Projections are versioned and independently positioned.
• Side-effecting behaviour is separated from derivation so replay cannot re-notify or re-enqueue.
• A comparison mode reports divergence between rebuilt and live projections before switchover.
• Switchover is atomic and reversible.
• Event schema versioning and upcasting are supported, with a worked example of an evolved event.
• Rebuild progress and estimated completion are observable.
• The inflated totals described in Cross-cutting: Read-model donation totals are inflated by the historical double-counting bug and need reconciliation from the event stream #368 are recomputed using this mechanism, with a before-and-after diff recorded.
• A rebuild of representative data completes within a documented time budget.

Scope

Roughly 5,000–7,000 lines, including tests.

Relevant files

• backend/src/eventSourcing/projections.js
• backend/src/eventSourcing/eventStore.js
• backend/src/db/schema.sql

Related

Provides the mechanism #368 needs.