`SortformerModules.streaming_update_async`: first-compression `spkcache_preds` seeding branch is unreachable — cache retains stale pop-time predictions, amplified into phantom speakers by `strong_boost`

[CaKTono]

Component: nemo/collections/asr/modules/sortformer_modules.py (observed in nemo-toolkit 2.7.3; verified present on current main at the time of writing: same torch.full(..., 0.0, ...) allocation, same >= 0 sentinel, no seeding flag in the state class)

Summary

In streaming diarization with async_streaming=True, the branch that is
supposed to seed spkcache_preds with the current forward's re-predictions at
the first speaker-cache compression can never execute. As a result the async
updater permanently scores cached frames with stale pop-time posteriors —
computed when the speaker cache was empty, i.e. the model's least-informed
predictions of the session — where the synchronous updater
(streaming_update) re-scores them with the current forward at seeding time.
Downstream, _compress_spkcache's strong_boost reserves cache capacity per
speaker column, so a handful of stale, marginal (>0.5) posteriors on an unused
slot can lock in a phantom speaker that the sync updater never produces on
identical input.

Mechanism

1. streaming_update_async reallocates its scratch buffer every call:
   updated_spkcache_preds = torch.full((B, ..., n_spk), 0.0, ...)
   and copies the live state (also zero-initialized at
   init_streaming_state) into it.
2. The seeding sentinel then tests
   if updated_spkcache_preds[batch_index, 0, 0] >= 0: — intended to mean
   "already compressed at least once". Sigmoid posteriors are non-negative
   and disabled frames are written as 0.0, so this condition is always
   true, and the elif first-compression branch
   (updated_spkcache_preds[b, :spkcache_len] = preds[b, :spkcache_len])
   is unreachable.
3. Consequently the cache keeps the posteriors recorded at FIFO-pop time
   (pop_out_preds from the first pop, when spkcache was empty). The
   sync updater instead seeds the accumulated cache from the current
   forward at the first compression
   (spkcache_preds = cat([preds[:, :spkcache_len], pop_out_preds])).
4. In _compress_spkcache, admission requires a posterior > 0.5
   (_disable_low_scores) and strong_boost adds ~+1.39 to the top-k
   frames of each speaker column independently — so a few stale frames
   marginally above 0.5 on an otherwise-unused slot are boosted into the
   kept set, self-reinforce on subsequent forwards (the model attends over
   cache frames labeled with that slot), and lock in a phantom speaker.

Reproduction / evidence

• Synthetic unit regime (hand-crafted preds exercising strong_boost):
  default async vs sync spkcache_preds diverge with max abs difference
  ~0.90; forcing the seeding branch to execute makes async == sync exactly
  (0.0).
• Real audio, batch size 1, fp32, streaming (0.96 s chunks, default
  geometry spkcache_len=188, fifo_len=188, spkcache_update_period=144):
  on a 3-speaker meeting excerpt (public AMI corpus material), sync and
  async first diverge at exactly the first forward after the first
  compression (chunk 28 with this geometry; the arithmetic predicts pops at
  chunks 15 and 27, first compression at 27). Async goes on to commit a
  4th speaker slot (52 chunks of slot-4 activity; ~64 s of one real
  participant relabeled as a new speaker) while sync stays at 3 speakers on
  bit-identical input features.
• Fix validation: patching only the seeding semantics (see below) makes
  async's posterior stream bit-identical to sync at all 187 chunks of
  the same session in fp32, and eliminates the phantom speaker entirely.
  (Under bf16 autocast, exact equality is not expected — the async layout
  executes a different op sequence — and we observe small residual numeric
  divergence unrelated to this bug.)

Suggested fix

Replace the sign-sentinel with an explicit per-row spkcache_seeded: bool
(or equivalent) in the streaming state, and on the first compression seed
updated_spkcache_preds[b, :spkcache_len] from the current preds exactly
as streaming_update does. A minimal proof-of-concept that restores sync
semantics is to change the scratch-buffer fill value from 0.0 to -1.0
(making the sentinel test meaningful); we validated that variant as
described above, but an explicit flag is the robust form.

Related, lower-priority observation while reading the same function: the
per-row pop length uses the batch-wide max_chunk_len
(pop_out_len = max(pop_out_len, max_chunk_len - max_fifo_len + fifo_len))
where the sync path uses the row's own chunk length — inert for
uniform-length batches, but a latent cross-row coupling for ragged ones.

Happy to provide the standalone reproduction scripts and the patch as a PR
if useful.

[ipmedenn]

Hi @CaKTono, this is a real bug, thanks for catching!
Will fix it soon.