Skip to content

FlushTracker stall: consumer killed without terminate callback #3981

Closed

Description

@alco

Parent: #3980

Scenario

A consumer process is killed in a way that bypasses the terminate/2 callback:

  • :kill signal: Even with Process.flag(:trap_exit, true) (consumer.ex:108), a :kill signal terminates the process immediately without calling terminate. From Erlang docs: "If the process receives a kill signal, it terminates, regardless of the trap_exit flag."
  • :brutal_kill supervisor shutdown: If a supervisor is configured with shutdown: :brutal_kill, child processes receive :kill.

What happens

  1. A transaction arrives. ShapeLogCollector.publish → ConsumerRegistry.publish → broadcast delivers the event to the consumer.
  2. The consumer processes the event, replies :ok. FlushTracker.handle_txn_fragment records the shape in last_flushed and min_incomplete_flush_tree.
  3. The consumer is killed (:kill signal, etc.) before the storage flush callback fires and notify_flushed is called.
  4. terminate/2 does NOT run. No cleanup happens. No handle_writer_termination, no remove_shape_async, no FlushTracker.handle_shape_removed.
  5. The shape's entry in FlushTracker becomes the permanent minimum, blocking last_global_flushed_offset from advancing.

Fix

This scenario can only be addressed by an active detection mechanism in ShapeLogCollector (see parent issue #3980). The terminate callback path is insufficient by definition — no amount of improvement to terminate or handle_writer_termination can help when the callback doesn't execute.

Activity

  1. changed the title [-]FlushTracker stall: consumer killed without terminate callback (OOM, :kill signal)[/-] [+]FlushTracker stall: consumer killed without terminate callback[/+] on Mar 16, 2026
  2. alco commented on Mar 17, 2026

    @alco
    ContributorAuthor

    Closing as non-issue:

    • we don't have explicit kills in the code, so this option is ruled out
    • consumer supervisor shutting down will also take the shape log collector with it, so the whole shapes subsystem goes down and no replication streaming can go on
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions