No Streaming API for streaming sortformer?

[Jacob-Bishop]

Is your feature request related to a problem? Please describe.

I would like an easy way to use the streaming sortformer on a stream of audio a chunk at a time.

Describe the solution you'd like

diarize_step() and reset() methods.

Describe alternatives you've considered

It is likely currently possible to use streaming_forward_step() to cobble something together, but it would be nice to have a high-level api like the diarize() method. [Edit: Unclear how to stream audio through the preprocessor].

Additional context

Add any other context or screenshots about the feature request here.

[Jacob-Bishop]

There's a tutorial here https://github.com/NVIDIA-NeMo/Speech/blob/main/tutorials/speaker_tasks/Streaming_End_to_End_Diarization_Inference.ipynb about using forward_streaming_step().

[Jacob-Bishop]

As far as I can tell, there isn't a way to do pre-processing on a streaming basis? The "streaming" tutorial pre-processes the audio, and then streams the features.

[rehan243]

Oh yeah, I ran into this exact thing. That bit in your edit about being unclear how to stream audio through the preprocessor, that's the tricky part. The preprocessors in NeMo usually expect the whole input upfront, so streaming isn't super straightforward.

One workaround I used was to manually chunk the audio and pass each chunk through AudioToMelSpectrogramPreprocessor before feeding it into streaming_forward_step(). Something like this:

preprocessor = nemo.collections.asr.modules.AudioToMelSpectrogramPreprocessor()
for chunk in audio_stream:
    mel = preprocessor(chunk)
    output = model.streaming_forward_step(mel)
    # Do something with output...

Not ideal, but it works if you manage chunk overlap carefully. Curious, are you trying to keep latency super low, or is this more about simplifying the API?

[Jacob-Bishop]

@rehan243 It wasn't clear to me whether preprocessor() had any normalization or padding that would be inappropriate in a streaming situation? Was thinking of using it for barge-in detection that would be more resistant to background noise.