AWS STT plugin extremely slow when using multi-user transcription

[pabloFuente]

Bug Description

When changing the plugin in the example agent multi-user-transcriber.py from deepgram to aws, the transcription starts getting very slow with 3 or more participants connected to the same room. When transcribing different participants from different Rooms, transcriptions behave well and arrive within a reasonable time.

This does not happen for any of the other 17 STT plugins. Only the Amazon Transcribe plugin behaves this way in this sample agent.

The agent log shows this warning message again and again:

{"message": "inference is slower than realtime", "level": "WARNING", "name": "livekit.plugins.silero", "delay": 13.077118384041238, "pid": 52, "job_id": "AJ_zzYdH8TPndaD", "timestamp": "2025-10-28T19:12:35.291028+00:00"}
{"message": "inference is slower than realtime", "level": "WARNING", "name": "livekit.plugins.silero", "delay": 12.733135948984998, "pid": 52, "job_id": "AJ_zzYdH8TPndaD", "timestamp": "2025-10-28T19:12:35.293962+00:00"}
{"message": "inference is slower than realtime", "level": "WARNING", "name": "livekit.plugins.silero", "delay": 13.187841181981316, "pid": 52, "job_id": "AJ_zzYdH8TPndaD", "timestamp": "2025-10-28T19:12:35.457624+00:00"}
{"message": "inference is slower than realtime", "level": "WARNING", "name": "livekit.plugins.silero", "delay": 13.44968232001003, "pid": 52, "job_id": "AJ_zzYdH8TPndaD", "timestamp": "2025-10-28T19:12:35.458104+00:00"}
{"message": "inference is slower than realtime", "level": "WARNING", "name": "livekit.plugins.silero", "delay": 13.052407222982552, "pid": 52, "job_id": "AJ_zzYdH8TPndaD", "timestamp": "2025-10-28T19:12:35.645356+00:00"}
{"message": "inference is slower than realtime", "level": "WARNING", "name": "livekit.plugins.silero", "delay": 13.400238036039632, "pid": 52, "job_id": "AJ_zzYdH8TPndaD", "timestamp": "2025-10-28T19:12:35.648003+00:00"}
{"message": "inference is slower than realtime", "level": "WARNING", "name": "livekit.plugins.silero", "delay": 13.929592445012062, "pid": 52, "job_id": "AJ_zzYdH8TPndaD", "timestamp": "2025-10-28T19:12:35.970727+00:00"}
{"message": "inference is slower than realtime", "level": "WARNING", "name": "livekit.plugins.silero", "delay": 13.66886186598532, "pid": 52, "job_id": "AJ_zzYdH8TPndaD", "timestamp": "2025-10-28T19:12:35.970937+00:00"}
{"message": "inference is slower than realtime", "level": "WARNING", "name": "livekit.plugins.silero", "delay": 14.252598414036559, "pid": 52, "job_id": "AJ_zzYdH8TPndaD", "timestamp": "2025-10-28T19:12:36.532489+00:00"}
{"message": "inference is slower than realtime", "level": "WARNING", "name": "livekit.plugins.silero", "delay": 13.905491034980454, "pid": 52, "job_id": "AJ_zzYdH8TPndaD", "timestamp": "2025-10-28T19:12:36.532791+00:00"}

Which I think is just the silero plugin complaining about delays in the pipeline. To me it seems that the AWS STT plugin is somehow saturating some kind of buffer when sending audio to Amazon Transcribe or receiving its events. But again, this does not happen for any other STT plugin.

Expected Behavior

The aws STT plugin should work fine in the multi-user-transcriner.py sample agent, just as well as with any other of the STT plugins.

Reproduction Steps

I have a very simple setup to demonstrate the problem at https://github.com/OpenVidu/livekit-agents-transcription-test/tree/multi-user
As its README states:

First of all clone the sample repository (be aware of checking out to branch multi-user):

git clone -b multi-user https://github.com/OpenVidu/livekit-agents-transcription-test.git
cd livekit-agents-transcription-test

Then:

1. Build the agent container

   docker build -t livekit/transcription-agent-test:latest agent/.

2. Export requried env vars and start the agent

   # Export your LiveKit Cloud credentials
   export LIVEKIT_URL=wss://xxxxxxxx.livekit.cloud
   export LIVEKIT_API_KEY=your_livekit_cloud_api_key
   export LIVEKIT_API_SECRET=your_livekit_cloud_api_secret
   
   # Export your AWS credentials
   export AWS_ACCESS_KEY_ID=your_access_key_id
   export AWS_SECRET_ACCESS_KEY=your_secret_access_key
   export AWS_DEFAULT_REGION=your_aws_region
   
   # Start the agent
   docker compose up -d

3. Set up your LiveKit Cloud credentials in the webapp HTML right here:

   const LIVEKIT_URL = "wss://xxxxxxxx.livekit.cloud";
   const LIVEKIT_API_KEY = "your_livekit_cloud_api_key";
   const LIVEKIT_API_SECRET = "your_livekit_cloud_api_secret";

4. Run the webapp

   cd webapp
   npm install
   npm start

5. Open http://localhost:3000 in your browser and test.

When adding 3 or more participants to the same room, speech events will be very delayed, becoming unusable:

```

Operating System

Linux

Package Versions

livekit-agents>=1.2.14

[tinalenguyen]

Hi, thanks for the detailed report @pabloFuente!

I looked through AWS Transcribe docs, turns out they don't currently support audio with more than two channels (source). I believe this would explain the latency for 3+ participants in the same room.

I was able to reproduce the logs you mentioned with 3 participants, however my logged latencies never surpassed 1 second. I wonder if that has to do with your AWS Region configuration

[pabloFuente]

I am sorry for commenting again, but I don't think this issue is solved.

You might think that 1 second latency is good enough for 3 participants, but the reality is that for 4 participants the delay becomes +10 seconds and for 5 transcription some messages take +1 minute to arrive. So the multi-user-transcriber.py becomes unusable. You can try it for yourself using my sample app.

I don't think the problem is in the number of audio channels supported by AWS Transcribe in a single connection. That would apply if the agent was somehow trying to multiplex all audio tracks in a single connection to AWS Transcribe. But the multi-user-transcriber.py should be creating different connections to AWS Transcribe, each one with 1 audio channel. Beside, hitting the 2-audio channel per-connection limit in AWS would surely trigger a fatal error, not a delay.

I have just done some research on the AWS Transcribe SDK and I have found this interesting issue: awslabs/amazon-transcribe-streaming-sdk#121. This is clearly related to this problem.

AWS is deprecating libary amazon-transcribe, which has some critical issues regarding precisely CPU bottlnecks when handling streaming. It is being replaced it with a new one called aws_sdk_transcribe_streaming, which means replacing this dependency https://github.com/livekit/agents/blob/main/livekit-plugins/livekit-plugins-aws/pyproject.toml#L28 with "aws_sdk_transcribe_streaming>=0.2.0". That library is already available at pypi: https://pypi.org/project/aws_sdk_transcribe_streaming/

I have tested this same scenario with the new AWS SDK, and the problem is virtually gone. Using the new official AWS Transcribe SDK just fixes the problem.

I have opened a PR updating the library. You are welcome to test the sample app using the livekit-plugin-aws of my PR and check on your own that the issue no longer appears: #4111