Skip to content

fix(engine): #872 lock-free memtable byte accounting for the resource sampler - #883

Open
MavenRain wants to merge 1 commit into
xerj-org:mainfrom
MavenRain:fix/872-memtable-bytes-accounting
Open

fix(engine): #872 lock-free memtable byte accounting for the resource sampler#883
MavenRain wants to merge 1 commit into
xerj-org:mainfrom
MavenRain:fix/872-memtable-bytes-accounting

Conversation

@MavenRain

@MavenRain MavenRain commented Aug 30, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes #872. The resource sampler's per-index size_bytes() no longer
read-locks all 16 memtable shards; it is now one Relaxed load of a
shared Arc<AtomicUsize> aggregate that every shard maintains
incrementally at each total_bytes write.

Root cause

Sampler thread B ("xerj-memtable-sampler", spawned in
engine.rs:3820 spawn_resource_sampler, comment hunk at
engine.rs:3878-3885) calls total_memtable_bytes() every
SAMPLE_INTERVAL_MS = 100 ms. That summed
ShardedFtsMemtable::size_bytes() for every index, and the old body
(memtable.rs, pre-fix line ~841) was
self.shards.iter().map(|s| s.read().size_bytes()).sum(): a read lock
on each of the 16 shards. At 464 idle indices that is
464 x 16 x 10 = ~74,240 lock acquisitions per second of pure idle
overhead, and a held shard write lock stalled the sampler outright.

Fix design

  • One Arc<AtomicUsize> per index, created in
    with_registry_and_shards (memtable.rs:789-805) and cloned into
    every FtsMemtable shard. ShardedFtsMemtable::size_bytes()
    (memtable.rs:854) is a single Relaxed load, no locks.
  • The atomic lives inside each FtsMemtable, not in the sharded
    wrapper, because with_shard_mut hands callers a raw
    &mut FtsMemtable: turbo ingest calls insert_raw_bytes_fresh
    directly through it, so wrapper-level accounting would miss that
    bypass entirely.
  • All 11 total_bytes write sites mirror the observed delta
    (insert_analyzed, insert_raw_bytes_with_seq,
    insert_raw_bytes_fresh, insert_pretokenized_inner,
    insert_pretokenized_arc, min-clamped remove, the four full-reset
    drains, and drain_shard_inner). Observed-delta mirroring means a
    reset subtracts exactly what the shard still carries, so nested or
    repeated resets contribute 0 and drains stay idempotent-safe.
  • AtomicUsize, not the issue's AtomicU64: total_bytes is usize,
    the width is the same on 64-bit targets, and it avoids a cast at
    every mirror site. Standalone FtsMemtable::new()/with_registry()
    get a private counter; FtsMemtable::size_bytes() is unchanged.

Breaker unaffected

The governor admission path reads its own relaxed atomics
(governor.rs:120-143 check_ingest_admission) and is untouched.
memtable_budget_trips_on_refresh and
build_from_default_config_trips_nothing stay green in the gate
battery.

Tests

  • size_bytes_does_not_take_shard_locks: proves size_bytes() returns
    while a shard write lock is held. The holder self-releases on a 4 s
    bound, so a lock-summing regression fails the 2 s receive timeout
    cleanly instead of deadlocking the test.
  • aggregate_bytes_matches_full_recount_after_every_mutator: proves
    the aggregate equals a full per-shard recount after every mutator
    (inserts of all four kinds, the with_shard_mut raw bypass, remove
    of an existing and a missing doc, every drain variant), ending at 0.
  • Mutation-confirm: M1 reverts size_bytes() to the lock-summing body;
    the lock test kills it. M2 zeroes the fetch_add(size, ...) mirrors;
    the recount test kills it. Both restores verified byte-clean against
    the staged fix.

Verification

Verified: the gate battery results in
~/Documents/xerj-872-gate.log: cargo fmt --check, build,
clippy --no-deps, the two new tests, the two governor tests,
mutation-confirm x2 with clean restores, and the ES-compat YAML suite.

Assumed (not measured): the production sampler CPU reduction at 464
indices. The ~74,240 locks/s figure is arithmetic from the constants in
#872, not a live-cluster measurement.

Closes #872

The resource sampler thread ("xerj-memtable-sampler", spawned in
engine.rs spawn_resource_sampler) calls total_memtable_bytes() every
SAMPLE_INTERVAL_MS=100 ms. That summed ShardedFtsMemtable::size_bytes()
per index, and size_bytes() read-locked every one of the 16 shards. At
464 idle indices that is 464 x 16 x 10 = ~74,240 shard lock
acquisitions per second of pure idle overhead, and any held shard write
lock stalled the sampler for the duration of the hold.

Fix: every FtsMemtable shard of one index now shares a single
Arc<AtomicUsize> aggregate with its ShardedFtsMemtable. All 11
total_bytes write sites mirror the observed delta into the atomic:
insert_analyzed, insert_raw_bytes_with_seq, insert_raw_bytes_fresh,
insert_pretokenized_inner, insert_pretokenized_arc, remove (min-clamped
like total_bytes itself), the four full-reset drains
(drain_with_sources_raw, drain_raw, drain, drain_with_sources), and
drain_shard_inner. Mirroring at the FtsMemtable level keeps the
accounting correct on the with_shard_mut raw-insert bypass that turbo
ingest uses. ShardedFtsMemtable::size_bytes() is now one Relaxed load
and takes no locks. A standalone FtsMemtable (new / with_registry)
keeps its own private counter and its size_bytes() is unchanged.

Note: the issue sketch names AtomicU64; the implementation uses
AtomicUsize because total_bytes is usize. Same width on 64-bit targets,
and it avoids a cast at every mirror site.

Tests added: size_bytes_does_not_take_shard_locks (size_bytes returns
while a shard write lock is held; the holder self-releases on a bound
so a lock-summing regression fails instead of hanging) and
aggregate_bytes_matches_full_recount_after_every_mutator (the aggregate
equals a full per-shard recount after every mutator, ending at zero).

Gates run: cargo fmt --check, build, clippy --no-deps, the new tests,
the governor breaker tests (memtable_budget_trips_on_refresh,
build_from_default_config_trips_nothing), mutation-confirm x2 (the
lock-summing revert and a dropped mirror are both caught), and the
ES-compat YAML suite.

Closes xerj-org#872
@cla-bot cla-bot Bot added the cla-signed label Aug 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Memtable sampler walks every index × every shard lock each 100 ms — one thread burns 12% of a core at 464 idle indices

1 participant