Skip to content

feat(placement): dual-route moving groups through existing outputs - #491

Draft
timtofan wants to merge 6 commits into
masterfrom
atimofieiev/placement-routing
Draft

timtofan wants to merge 6 commits into
masterfrom
atimofieiev/placement-routing

Conversation

@timtofan

@timtofan timtofan commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Why

Moving a shard currently changes ownership derived from the host list. Explicit placement groups let one group move while preserving every other name's destination. During a move, live writes must reach both the donor and destination while immutable history is copied. Reapplying a point is safe under the storage contract that (name, slot) writes are idempotent and rollups are recomputed from slots.

Architecture

An opt-in placement cluster loads the canonical virtual-shard map and binds its store incarnations to the existing configured host endpoints. Frozen bootstrap selectors preserve existing jump owners. The scalar and batch paths select the replica's writer and optional moving_to from one immutable map snapshot, then use ordinary bounded host queues and existing transports.

SIGHUP validates a complete successor and publishes it with one atomic pointer swap per cluster. Validation checks the predecessor digest, revision, frozen identity, monotone write epochs and fixed transport inventory. A writer change must promote the prior moving_to and clear it. Failed reloads retain the active map and increment a rejection counter. A process restart reads the configured current map at any revision.

Relay output carries ordinary points through the existing queues; placement epochs remain control metadata. Store-side snapshot pinning, checksum copies, copied-part precedence, read switching, relay drain and final incremental copying belong to the move controller. A final copy can fill missing slots but cannot repair a dropped same-slot overwrite when B already has an older live-written value, because copied parts must remain older than B's live parts. Already staged output can finish under the preceding map, so adoption alone does not certify drain.

moving_to is an optional map extension. Omitting it preserves existing canonical bytes and golden digests. Older strict decoders reject documents using it; store/controller schema support must be available before moving maps are enabled. The copied contract's historical receipt helpers remain unused by relay routing.

Before / after

Before After
Routing derives destinations from a jump ring or broadcasts to all hosts Default jump routing stays intact; opt-in placement selects stable groups and explicit store IDs
Changing host count can move unrelated names A successor changes only the selected groups' owners
There is one selected destination per replica A moving group sends each point to both its donor and declared destination
Reload cannot express a staged ownership move SIGHUP adopts dual routing, then a successor promotes the destination at the next write epoch
Existing output uses bounded in-memory queues Placement reuses those same queues and their drop counters; copying retains older-part precedence; a dropped same-slot overwrite requires store-side handling

Usage

Configure type = "placement", placement_map, placement_replica, and explicit hosts as shown in docs/placement-routing.md. The shared bootstrap example is exercised by scalar/batch legacy equality tests.

For one replica, publish a successor containing this fragment:

{"writer":"a","moving_to":"d","write_epoch":1}

After snapshot/incremental copying and the read switch, publish its successor:

{"writer":"d","write_epoch":2}

Wait for all relays to adopt and drain, perform the final incremental copy, then retain the donor until collection is safe. A group carrying byte share f adds one extra logical copy of its records, increasing logical egress from B to B*(1+f); actual compressed NIC traffic also depends on framing and dictionaries.

Validation

  • Local Go 1.27 full build, vet, normal tests and race suite pass. The unchanged shutdown integration test passed targeted repeats and a full recheck after an initial startup-readiness failure.
  • Both shared JSON fixtures are byte-identical to the source contract; golden routing vectors and canonical map digest pass.
  • Scalar and actual batch staging cover dual delivery, final promotion, restart, stale/invalid rejection and concurrent adoption. Warmed routing and dispatch use 0 B and 0 allocations per record.
  • One synthetic moving group adds 2,504 deliveries per 10,000 inputs (+25.04%) and 25.05894% logical plaintext bytes. General extra logical egress equals the moving group's byte share.

Warmed routing/dispatch on a shared Apple M4 Max laptop, GOMAXPROCS=1, five 300ms repetitions with counting targets:

Path ns/record M records/s Allocs/record
single/batch=false 46.69 21.42 0
single/batch=true 50.69 19.73 0
one-moving/batch=false 46.89 21.33 0
one-moving/batch=true 49.68 20.13 0
all-moving/batch=false 45.92 21.78 0
all-moving/batch=true 51.98 19.24 0

These figures exclude parsing, output queue locks, encoding, sockets and stores; they do not establish NIC throughput.

The final paired legacy comparison uses 31 processes, each with 2,048 alternating windows of 4,096 records per layout: hosts=3/batch=False: -4.96% (95% process-bootstrap -6.94% to -4.05%); hosts=3/batch=True: +0.17% (95% process-bootstrap +0.16% to +0.34%); hosts=40/batch=False: -2.21% (95% process-bootstrap -2.44% to -1.48%); hosts=40/batch=True: +1.58% (95% process-bootstrap +0.90% to +3.45%). A 40-host batch control comparing two copies of the baseline reports +1.58% (95% process-bootstrap +0.09% to +3.15%). This demonstrates sensitivity to the benchmark/code layout on the shared laptop. The requested zero-regression qualification remains open; neither source equality nor this control is treated as proof of zero CPU cost.

Rollout and rollback

Keep this PR Draft until the matching store/controller move implementation and schema support are ready. Provision all map endpoints before a move; map reloads cannot edit transport inventory. Preserve the current map through restarts. Reverse an ownership move through successive maps and epochs with store reconciliation rather than restoring an old file.

@timtofan timtofan changed the title feat(routing): add opt-in placement maps and epoch barrier API feat(placement): persist epoch delivery through durable acknowledgements Nov 16, 2026
@timtofan timtofan changed the title feat(placement): persist epoch delivery through durable acknowledgements feat(placement): add placement routing and durable epoch output Nov 16, 2026
@timtofan timtofan changed the title feat(placement): add placement routing and durable epoch output feat(placement): dual-route moving groups through existing outputs Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant