Add inference bargein and examples

[chenghao-mou]

• New barge-in detector under inference
• Two stream implementation:
  • HTTP endpoints for working with hosted model
  • WS for working with gateway proxy

Detailed spec can be found in the notion page

[theomonnom]

Nice work!

[theomonnom]

Let's move this file to other

[chenghao-mou]

I will remove it before merging. I don't think the user will need it.

[theomonnom]

Can it just be:

Suggested change

	from livekit.agents.inference.bargein import BargeinDetector
	from livekit.agents.inference import BargeinDetector

[theomonnom]

I like this pattern (it follows how we use STT/TTS in the inference gateway in our docs)

Suggested change

	bargein_detector=BargeinDetector(),
	bargein_detector=inference.BargeinDetector(),

Tho I'm also wondering if it should always be there by default.

[theomonnom]

I see that we're only re-emitting events inside _transcript_buffer when we receive a new STT event. Shouldn't we also re-emit it if we detect it was a bargin? The STT may not send us any additional events, so there is a case where we just ignore this buffered event.

[chenghao-mou]

Yes, I don't have a return in this elif branch. So that after-the-barge-in event will be processed normally.

[theomonnom]

What I mean is what if this scenario happens:

1. should_hold_event is True
2. we buffer the transcript
3. we detected it was indeed a barge-in
4. we never receive new transcripts from the STT

In this case it seems like we just lose the buffered transcript and we don't generate any new generation?

[chenghao-mou]

Great question! Here is a breakdown of what I am thinking when there is no new transcript:

Scenario 1: If there is no actual user speech/barge-in

This is a barge-in false positive. If resume_false_interruption is enabled, the audio will pause and then resume. If not enabled, it will interrupt with no new transcript.

Scenario 2: If there is actual user speech/barge-in but no new transcript

Case 1: This is an STT failure. But it should behave similarly to Scenario 1.

Case 2: The transcript for the barge-in comes before the inference is done

The typical audio needed for the model for a barge-in detection is around 300ms. This means we can lose the transcript for that 300ms window. I think it is very rare to have a true barge-in for just 300ms of speech and STT finishes transcribing that before it's detected.

But there is a non-zero probability of barge-in detection being late, in which case, we might lose more transcript. What I can do here is to re-emit transcripts back until the last speaking point (if we have the timing information), but it might include non-barge-in transcript if the user says something like "Right, right [bc], we should ...[barge-in]".

[chenghao-mou]

Okay, I have updated it so that it will flush either at the end of agent speech or at new STT events up to the last overlap speech start.