Repository navigation
FlushTracker has no active mechanism to detect shapes that stop reporting flush progress #3980
Description
Summary
The FlushTracker in ShapeLogCollector is a passive data structure that relies entirely on external notifications to advance last_global_flushed_offset. When a shape enters tracking via handle_txn_fragment, the tracker expects one of two things to eventually happen:
handle_flush_notification— the consumer reports it flushed the offsethandle_shape_removed— the shape is cleaned up
If neither happens, that shape's entry becomes the minimum in min_incomplete_flush_tree and permanently blocks WAL flush advancement for the entire stack.
ShapeLogCollector does not monitor consumer processes. There is no timeout, no periodic sweep, and no liveness check for shapes tracked by FlushTracker. The system is entirely dependent on every consumer eventually calling notify_flushed or being properly removed — a property that is not guaranteed across all failure modes.
Impact
When last_global_flushed_offset stops advancing, the replication client never receives {:flush_boundary_updated, lsn}, Postgres never gets WAL flush acknowledgement past that point, and WAL accumulates indefinitely until disk is exhausted.
What's already addressed
- fix(sync-service): recover from dead consumers blocking WAL flush advancement #3975 fixes the case where a consumer dies during
ConsumerRegistry.broadcast— the dead consumer is detected via:DOWN, cleaned from ETS, and a replacement is started. Events for shapes that were removed between routing and delivery are excluded from FlushTracker entirely.
What remains unaddressed
The child issues below describe specific failure modes that can leave shapes permanently tracked in FlushTracker with no path to resolution.
Suggested fix
Add a defense-in-depth mechanism in ShapeLogCollector to actively detect when FlushTracker-tracked shapes have stopped progressing. Two approaches:
Option A: Monitor consumer PIDs. When ConsumerRegistry.publish delivers events, capture the consumer PIDs. ShapeLogCollector monitors them. On :DOWN, call FlushTracker.handle_shape_removed. This is fast (immediate detection) but requires ShapeLogCollector to handle :DOWN messages and maintain a PID→shape mapping.
Option B: Periodic liveness sweep. On each incoming transaction (or on a timer), ShapeLogCollector iterates FlushTracker.last_flushed and checks whether each shape still has a live consumer via ConsumerRegistry.whereis + Process.alive?. Dead shapes get handle_shape_removed. This is simpler but detection is delayed until the next sweep.
Option A catches all crash/kill scenarios immediately. Option B also catches the "alive but stuck" scenario (child issue #3) if extended with a staleness timeout.
Activity
msfstef commentedon Mar 9, 2026
On Option A - we had moved to the request batcher to avoid flooding the shape log collector with shape registration/lifecycle events since its main job is to handle the transactions and push to the right consumers. I'd be wary about reintroducing that. Maybe the flush tracker should not live on the shape log collector, and should be its own process?
alco commentedon Mar 9, 2026
@msfstef The issue description is not telling the full story. It focuses on ShapeLogCollector alone, but it calls
ConsumerRegistry.publish(), which in turn callsbroadcast()and that function monitors every consumer it sends the txn fragment to, as is the standardGenServer.call()protocol.In any case, I think the actual approach for resolving the problem is somewhere between #3975 and #3979
alco commentedon Mar 17, 2026
Proposed approach: Periodic FlushTracker liveness sweep (Option B)
Rather than monitoring consumer PIDs in ShapeLogCollector (Option A), add a periodic sweep that detects when
last_global_flushed_offsethas been stuck for too long and checks the liveness of the consumers responsible for blocking it.Design
FlushTracker gains a single query function
blocking_shape_ids/1that returns the shape IDs at the smallest position inmin_incomplete_flush_tree— these are the shapes currently preventing the global flush offset from advancing.ShapeLogCollector adds a recurring timer (default every 30s) with two-phase staleness detection:
last_global_flushed_offsetwith the previous sweep's cached valueWhen the threshold is exceeded, the sweep checks each blocking shape:
handle_shape_removeddirectlyConsumerRegistry.whereisreturns nil or dead PID) → triggerShapeCleaner.remove_shape_async, which invalidates the shape first (clients get 409), then cleans up FlushTracker through the standardremove_subscriptionpath. This guarantees no data loss — the FlushTracker entry is only removed after the shape is fully invalidated.A
shapes_pending_sweep_removalset prevents duplicateremove_shape_asynccalls across consecutive sweep ticks while waiting for the async cleanup pipeline to complete. Telemetry on[:electric, :flush_tracker, :sweep]reports counts of orphaned/dead/alive-stuck detections.Safety
ShapeCleaner.remove_shape_async→ShapeStatus.remove_shape→Consumer.stop→Storage.cleanup!→ShapeLogCollector.remove_shape→FlushTracker.handle_shape_removed.ShapeStatus.remove_shapeis idempotent,EventRouter.has_shape?guardsremove_subscription, andFlushTracker.handle_shape_removedon an unknown shape is a no-op.handle_writer_terminationremoves it). If the shape blocks FlushTracker for >60s with no new transactions arriving, the sweep will invalidate it. This is a deliberate trade-off — the cost of a client refetch is low compared to unbounded WAL accumulation.Primary catch scenarios
This sweep is defense-in-depth on top of the primary fix in #4011. Its two unique catches are:
kill -9terminate/2doesn't run, so no cleanup firesScope
~3 files modified:
flush_tracker.ex(addblocking_shape_ids/1),shape_log_collector.ex(sweep timer + handler),stack_config.ex(default config values). No changes to FlushTracker's data model — it remains a pure data structure.fix: Prevent FlushTracker stalling when consumer processes crash (#4011)
Address review feedback: clarify test intent and known-bug status
:flushednotification from WriteLoop to Consumer blocks WAL acknowledgment #4058fix: Prevent FlushTracker stalling when consumer processes crash (#4011)
mheisig commentedon May 12, 2026
Adding a data point — we hit this exact pattern in production with ElectricSQL 1.5.0 on 2026-05-08 → 2026-05-11.
Symptom: a single shape's row set silently diverged from Postgres for ~73 hours. Postgres had 13 rows, the materialised shape held 9 plus a deleted ghost. No errors anywhere; the replication slot was healthy with 3 KB lag; client snapshots returned
up-to-dateagainst a stale handle.Trigger: chronic
:system_memory_high_watermarkalarm flapping (66 set/clear transitions in 7 days, because Erlang'smemsupreads host/proc/meminforather than the container's cgroup limit — our 1 GB container limit didn't retarget the probe). One of those transitions stranded the consumer forProject where channel_id = '...'in an active-but-not-receiving state. The replication client kept advancing for other shapes, masking the issue.Why it stayed undetected:
confirmed_flush_lsnlooked healthy)up-to-dateresponses — Electric served the stale on-disk state and confidently told the client "you have everything"Recovery: required SSH into the host,
docker stop hitfactor-electric,rm -rf shapes/ meta/shape-db/, restart. The DELETE/v1/shapeAPI works onceELECTRIC_ENABLE_INTEGRATION_TESTING=trueis set, but doesn't help when you can't detect the drift in the first place.What we built downstream to compensate (in case it's useful as a "what the hole looks like to users" signal):
?offset=-1) and Postgres, hashes the sorted ID lists, and auto-rebuilds via DELETE on mismatchReinforces the Option A suggestion above: per-shape PID monitoring would have caught this. The shape's consumer process was alive but its WAL application loop had stopped — exactly the case "no timeout, no periodic sweep, no liveness check" describes.
Happy to share more details from the postmortem (we wrote it up internally) if it helps prioritisation.