[WIP][Model] Extend Ultravox to accept audio longer than 30s

[farzadab]

Currently the Ultravox model input is capped to 30 seconds and extra audio is truncated (AFAIK). Also each sample is fed to Whisper individually (without being batched).

This PR allows using longer audio by chunking them first, using Whisper encoder in batch mode, and then concatenates them.

TODO:

• processors on HF still need to be updated in tandem with this PR.
• run evaluations with the updated model to verify the changes.
• add test with long audio

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[farzadab]

This is temporary and will be reverted.

[farzadab]

Processor is updated to handle multiple audio. This is still a bit of a WIP since 1) not all models are updated, only the v0_3 version of 1B is updated, and 2) during this PR I realized that the new processor will break this VLLM implementation, so I have to figure out what to do there.

[farzadab]

I'm not sure if I'm doing the right thing here. I just want to say there's no limit to input audio anymore.

[mgoin]

FYI @NickLucche for the usage of whisper

[NickLucche]

Thanks for the contrib!
What is the chunking logic for tiling the audio? Feel free to link the hf processor PR.

[farzadab]

I might be doing something wrong here, perhaps I should do .flat_from_sizes but I'm not sure how that works closely.

[DarkLight1337]

You can use .flat_from_sizes in the processor to represent variable batch size per audio (assuming that the processor concatenates the result across each audio). The processor cache should still work correctly as long as the processor output for a given audio input doesn't depend on other audio inputs.

Inside the model, you can torch.split them according to audio_num_chunks to recover the original batches.

[farzadab]

re @NickLucche: Here's the processor link: https://huggingface.co/fixie-ai/ultravox-v0_3-llama-3_2-1b/blob/main/ultravox_processing.py#L209

The logic: for each audio, split to 30 second chunks (but do not pad the last item to 30s, which is the same as before).
Then we flatten and batch everything up and run Whisper as if they were separate audios. We use audio_lens to compute an attention_mask for the last chunk per audio. The final embeddings are then concatenated.

There are other ways we could've done this, but it matches what we do on the Ultravox side for both some fine-tuning that we do and evals. If we end up updating those I'll update VLLM as well.

Also, note that since we don't pad the last chunk, and since in most cases we have smaller than 30s audio, the number of frames do not match across samples. I didn't see a collator anywhere that I could update. I'm suspecting that I'll have to update _process_audio_input further to handle that. Updated _process_audio_input.

[NickLucche]

Ok I see then that's a naive chunking where you don't account for splitting mid-word nor you have any overlap and/or prompt from previous chunk.

This case seems much easier to handle vllm-side, given changes are already in hf. Let's just make sure the batched whisper forward is accounted for by the initial profiler run to avoid oom.