Repository navigation
fix(sync-service): promote adopted dependency consumers to transaction writes - #4780
Merged
Merged
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4780 +/- ##
==========================================
+ Coverage 60.05% 60.07% +0.01%
==========================================
Files 397 397
Lines 43772 43772
Branches 12590 12591 +1
==========================================
+ Hits 26289 26296 +7
+ Misses 17401 17394 -7
Partials 82 82
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
✅ Deploy Preview for electric-next ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
alco
force-pushed
the
fix/promote-inner-consumer-write-unit
branch
from
September 1, 2026 09:15
91fbe92 to
2b4c822
Compare
Base automatically changed from
fix/start-dependency-consumer-on-demand
to
main
September 1, 2026 09:24
alco
added a commit
that referenced
this pull request
Sep 1, 2026
…rent (#4779) This PR is based on the code from #4774 and supersedes it. ## Problem A subquery dependency can remain registered in ShapeStatus with a completed snapshot but no running consumer. When a new parent resolves its subquery to that existing handle, parent startup assumes the dependency consumer already exists. Its materializer fails to subscribe, the parent is invalidated, and every retry repeats the same failure because the dependency remains registered. One way to reach this state is for a dependency to become orphaned before restart. The restart pruning pass only recognizes dependencies through surviving parents, while restored standalone shapes start consumers lazily when transaction routing first needs them. ## Solution When ShapeCache resolves an existing shape while creating an inner shape, ensure its consumer is running before starting the parent. Missing consumers are restored through the same dependency-aware path used by transaction routing, while preserving `is_subquery_shape?: true` so the recovered dependency buffers complete transactions. The shared restoration helper also removes duplicated consumer-starting logic. ## Test The regression test constructs the registered/completed-snapshot/no-consumer state, requests a new parent, and verifies that: - the existing dependency handle is reused; - its consumer is restarted before the parent initializes; - the recovered dependency uses `write_unit: :txn`. Focused verification: `mix test test/electric/shape_cache_test.exs:1081`. ## Follow-up A pre-existing edge case remains when an already-running standalone consumer is later reused as a dependency: its write mode must be promoted at a transaction boundary. That behavior is addressed in [#4780](#4780). --------- Co-authored-by: Jacob Clark <jacob.clark@UK-JCLARK.local> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
`Consumer.subscribe_materializer/3` used the default 5-second `GenServer.call` timeout, unlike every other call on the materializer startup path (`await_snapshot_start/3`, `Materializer.subscribe/1`, `wait_until_ready/1`, `new_changes/3`), which already wait with `:infinity` and rely on process monitors for liveness. Bring it in line so the consumer is free to reply whenever it is actually safe to subscribe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016uMB7eVy1iXSfY8gqcn5A8
Deferring only the write-unit promotion was not enough. A materializer's
initial history read is bounded by storage's `last_seen_txn_offset`, which
advances only on commit, while a fragment-streaming consumer's
`latest_offset` advances with every fragment it writes. Replying to a
mid-transaction subscription with that offset let the materializer skip
past fragments it could not yet read, silently dropping the head of the
transaction from its index: a two-fragment transaction inserting rows 1
and 2 left the materializer with link values `[2]`.
Park the subscription instead: when `subscribe_materializer` arrives while
a fragmented transaction is pending, stash `{pid, from}` in
`deferred_materializer_subscription` and complete it (monitor, promote,
reply) from `finish_pending_txn/1` at the commit boundary. The consumer is
not marked as subscribed during the wait, so it never calls into the
blocked materializer; the materializer then reads a fully committed
history and receives every later change as a whole transaction.
`State.mark_materializer_subscribed/1` now assumes that invariant and
promotes directly; the separate `maybe_promote_write_unit/1` hook is gone.
A consumer already buffering whole transactions can still be subscribed
mid-transaction, since it has no partial writes on disk.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016uMB7eVy1iXSfY8gqcn5A8
alco
force-pushed
the
fix/promote-inner-consumer-write-unit
branch
from
September 1, 2026 09:29
2b4c822 to
873bb50
Compare
Contributor
|
This PR has been released! 🚀 The following packages include changes from this PR:
Thanks for contributing to Electric! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
Shape consumers without their own subquery dependencies start with
write_unit: :txn_fragment. Consumers known to be inner shapes start withwrite_unit: :txn, because dependency materializers and outer-shape evaluation require transaction-atomic updates.The startup hint is not sufficient when an already-running consumer is later adopted as a dependency. This can happen in two pre-existing sequences:
In both cases the dependency materializer can subscribe to a consumer that remains in fragment-write mode.
Fix
Treat materializer subscription as the authoritative signal that a consumer is an inner shape and promote it to whole-transaction writes.
If no transaction is in progress, promotion is immediate. If the subscription arrives during a fragmented transaction, that transaction finishes in fragment mode and promotion occurs at its commit boundary. This avoids mixing already-written fragments with a buffered tail of the same transaction.
The existing
is_subquery_shape?startup option remains a fast path for consumers known to be dependencies when they start.Tests
:txn_fragmentto:txn;110 passed, 1 excludedacross ShapeCache, consumer state, and consumer tests.Stack
This PR is stacked on #4779 and should be reviewed against
fix/start-dependency-consumer-on-demanduntil that PR merges.