You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
FlushTracker stall: consumer killed without terminate callback #3981
A consumer process is killed in a way that bypasses the terminate/2 callback:
:kill signal: Even with Process.flag(:trap_exit, true) (consumer.ex:108), a :kill signal terminates the process immediately without calling terminate. From Erlang docs: "If the process receives a kill signal, it terminates, regardless of the trap_exit flag."
:brutal_kill supervisor shutdown: If a supervisor is configured with shutdown: :brutal_kill, child processes receive :kill.
What happens
A transaction arrives. ShapeLogCollector.publish → ConsumerRegistry.publish → broadcast delivers the event to the consumer.
The consumer processes the event, replies :ok. FlushTracker.handle_txn_fragment records the shape in last_flushed and min_incomplete_flush_tree.
The consumer is killed (:kill signal, etc.) before the storage flush callback fires and notify_flushed is called.
terminate/2 does NOT run. No cleanup happens. No handle_writer_termination, no remove_shape_async, no FlushTracker.handle_shape_removed.
The shape's entry in FlushTracker becomes the permanent minimum, blocking last_global_flushed_offset from advancing.
Fix
This scenario can only be addressed by an active detection mechanism in ShapeLogCollector (see parent issue #3980). The terminate callback path is insufficient by definition — no amount of improvement to terminate or handle_writer_termination can help when the callback doesn't execute.
changed the title [-]FlushTracker stall: consumer killed without terminate callback (OOM, :kill signal)[/-][+]FlushTracker stall: consumer killed without terminate callback[/+]on Mar 16, 2026
we don't have explicit kills in the code, so this option is ruled out
consumer supervisor shutting down will also take the shape log collector with it, so the whole shapes subsystem goes down and no replication streaming can go on
Activity
[-]FlushTracker stall: consumer killed without terminate callback (OOM, :kill signal)[/-][+]FlushTracker stall: consumer killed without terminate callback[/+]alco commentedon Mar 17, 2026
Closing as non-issue: