Skip to content

Core updates for Streaming Sortformer inference #16032

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ipmedenn wants to merge 8 commits into
base: main
Choose a base branch
from
Open

Core updates for Streaming Sortformer inference #16032

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
798f9ef
Core updates for Streaming Sortformer inference
ipmedenn Aug 5, 2026
b3a2ba0
Addressing CodeQL findings
ipmedenn Aug 5, 2026
571717c
refactoring async mode
ipmedenn Aug 6, 2026
18beb17
updates for async mode
ipmedenn Aug 6, 2026
a111de8
fixing onnx export
ipmedenn Aug 6, 2026
dbc7724
adding pytest parametrize decorators to tests
ipmedenn Aug 7, 2026
a0d6602
minor updates in sortformer_modules.py
ipmedenn Aug 11, 2026
987e030
modularizing streaming_update_async method
ipmedenn Aug 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions ...eaker_tasks/diarization/conf/neural_diarizer/sortformer_diarizer_hybrid_loss_4spk-v1.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,10 @@ model:
sample_rate: 16000
pil_weight: 0.5 # Weight for Permutation Invariant Loss (PIL) used in training the Sortformer diarizer model
ats_weight: 0.5 # Weight for Arrival Time Sort (ATS) loss in training the Sortformer diarizer model
ats_tolerance: 0 # Arrival-time tolerance in output frames; tied speakers are matched by prediction overlap
max_num_of_spks: 4 # Maximum number of speakers per model; currently set to 4
high_resolution: false # If true, learn predictions at the preprocessor frame step (normally 10 ms)
output_subsampling_factor: ${model.encoder.subsampling_factor} # Returned prediction step in preprocessor frames

model_defaults:
fc_d_model: 512 # Hidden dimension size of the Fast-conformer Encoder
Expand Down Expand Up @@ -43,7 +46,7 @@ model:
bucket_buffer_size: 20000
shuffle_buffer_size: 10000
window_stride: ${model.preprocessor.window_stride}
subsampling_factor: ${model.encoder.subsampling_factor}
subsampling_factor: ${model.encoder.subsampling_factor} # Overridden by the model in high-resolution mode

validation_ds:
manifest_filepath: ???
Expand All @@ -65,7 +68,7 @@ model:
drop_last: False
pin_memory: True
window_stride: ${model.preprocessor.window_stride}
subsampling_factor: ${model.encoder.subsampling_factor}
subsampling_factor: ${model.encoder.subsampling_factor} # Overridden by the model in high-resolution mode

preprocessor:
_target_: nemo.collections.asr.modules.AudioToMelSpectrogramPreprocessor
Expand Down Expand Up @@ -162,7 +165,7 @@ trainer:
max_epochs: 800
max_steps: -1 # computed at runtime if not set
num_nodes: 1
strategy: ddp_find_unused_parameters_true # Could be "ddp"
strategy: ddp
accumulate_grad_batches: 1
deterministic: True
enable_checkpointing: False
Expand Down
12 changes: 8 additions & 4 deletions ...speaker_tasks/diarization/conf/neural_diarizer/streaming_sortformer_diarizer_4spk-v2.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,10 @@ model:
sample_rate: 16000
pil_weight: 0.5 # Weight for Permutation Invariant Loss (PIL) used in training the Sortformer diarizer model
ats_weight: 0.5 # Weight for Arrival Time Sort (ATS) loss in training the Sortformer diarizer model
ats_tolerance: 0 # Arrival-time tolerance in output frames; tied speakers are matched by prediction overlap
max_num_of_spks: 4 # Maximum number of speakers per model; currently set to 4
high_resolution: false # If true, learn predictions at the preprocessor frame step (normally 10 ms)
output_subsampling_factor: ${model.encoder.subsampling_factor} # Returned prediction step in preprocessor frames
streaming_mode: True

model_defaults:
Expand Down Expand Up @@ -44,7 +47,7 @@ model:
bucket_buffer_size: 20000
shuffle_buffer_size: 10000
window_stride: ${model.preprocessor.window_stride}
subsampling_factor: ${model.encoder.subsampling_factor}
subsampling_factor: ${model.encoder.subsampling_factor} # Overridden by the model in high-resolution mode

validation_ds:
manifest_filepath: ???
Expand All @@ -66,7 +69,7 @@ model:
drop_last: False
pin_memory: True
window_stride: ${model.preprocessor.window_stride}
subsampling_factor: ${model.encoder.subsampling_factor}
subsampling_factor: ${model.encoder.subsampling_factor} # Overridden by the model in high-resolution mode

test_ds:
manifest_filepath: null
Expand All @@ -89,7 +92,7 @@ model:
drop_last: False
pin_memory: True
window_stride: ${model.preprocessor.window_stride}
subsampling_factor: ${model.encoder.subsampling_factor}
subsampling_factor: ${model.encoder.subsampling_factor} # Overridden by the model in high-resolution mode

preprocessor:
_target_: nemo.collections.asr.modules.AudioToMelSpectrogramPreprocessor
Expand Down Expand Up @@ -123,6 +126,7 @@ model:
max_index: 99999 # Maximum allowed index value for internal processing in speaker cache update
scores_boost_latest: 0.05 # Gain for scores for recently added frames in speaker cache update
sil_threshold: 0.2 # Threshold for determining silence frames to calculate average silence embedding
use_learnable_sil_emb: false # Use a learned embedding instead of the running mean for silence cache slots
strong_boost_rate: 0.75 # Rate determining number of frames per speaker that receive strong score boosting
weak_boost_rate: 1.5 # Rate determining number of frames per speaker that receive weak score boosting
min_pos_scores_rate: 0.5 # Rate threshold for dropping overlapping frames when enough non-overlapping exist
Expand Down Expand Up @@ -206,7 +210,7 @@ trainer:
max_epochs: 800
max_steps: -1 # computed at runtime if not set
num_nodes: 1
strategy: ddp_find_unused_parameters_true # Could be "ddp"
strategy: ddp
accumulate_grad_batches: 1
deterministic: True
enable_checkpointing: False
Expand Down
Loading
Loading