Add Parakeet TDT 0.6B V3 batching example

[jiegong-fde]

Summary

Adds nvidia/parakeet-tdt-0_6b-v3-batching — a Truss for NVIDIA parakeet-tdt-0.6b-v3 English ASR with server-side dynamic batching for high-throughput HTTP transcription. It's the batching counterpart to the existing nvidia/parakeet-tdt-0_6b-v2 (single-request) example.

Details

• NeMo model run directly (auto dtype: fp16 on Turing/T4, bf16 on Ampere+).
• Pipelined batching: a collector thread drains the request queue (adaptive collect window) + runs the mel preprocessor while an inference thread runs encoder + TDT decoder (CUDA graphs) on the previous batch.
• Tunable via env vars: MAX_BATCH_SIZE, BATCH_COLLECT_MS, CUDA_GRAPHS, TORCH_COMPILE.
• Weights pulled from HF into a Baseten volume cache (model_cache); model is public (no token). Default accelerator L4.
• Accepts audio_url or audio_base64 (+ optional timestamps).

Test plan

• truss push deploys on L4 and serves {"transcript": ...} for the example audio_url.
• Concurrent requests batch on the GPU (batch size grows under load).

Made with Cursor

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}