feat(cli): add TwelveLabs Pegasus transcription backend - #6
Open
mohit-twelvelabs wants to merge 1 commit into
Open
feat(cli): add TwelveLabs Pegasus transcription backend#6mohit-twelvelabs wants to merge 1 commit into
mohit-twelvelabs wants to merge 1 commit into
Conversation
Add an opt-in 'pegasus' backend that transcribes a video directly from its URL using TwelveLabs Pegasus video understanding. Unlike the local and openai backends it reads the media server-side, so it skips the yt-dlp download and ffmpeg cleaning steps and uses the full audiovisual context. - New core/twelvelabs.ts wraps the v1.3 /analyze endpoint (stream:false) - Wire 'pegasus' into Backend, config defaults, validation, pipeline, and the transcribe command + JSON schema - URL input only; clear error for local files - Docs (getting-started, commands, configuration) updated - Tests: no-network dry-run/validation + key-gated live smoke test Requires TWELVELABS_API_KEY. Non-breaking: defaults and existing backends are untouched.
|
@mohit-twelvelabs is attempting to deploy a commit to the Crafter Station Team on Vercel. A member of the Team first needs to authorize it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi! I'm Mohit, I work at TwelveLabs (@mohit-twelvelabs).
What this adds
A new opt-in
pegasustranscription backend powered by TwelveLabs Pegasus video understanding. It sits alongside the existinglocal(whisper-cli) andopenaibackends:Unlike the other backends, Pegasus reads the video directly from its URL server-side — so it skips the
yt-dlpdownload andffmpegcleaning steps entirely and transcribes using the full audiovisual context (on-screen text, speaker cues), which can help on noisy or multi-speaker footage. Output is the usual.txtplus a single-cue.srt(Pegasus returns prose, not timed segments).Why it helps trx
trx is built around media-from-a-URL workflows, and that's exactly Pegasus' sweet spot: no local model download, no GPU, and the model sees the video, not just the audio. It's a natural third backend for the agent-first design — fully introspectable via
trx schema transcribe.Opt-in / non-breaking
localstays the default backend).local/openaipaths are untouched.-b pegasus; it requiresTWELVELABS_API_KEYand a direct media URL (clear error otherwise).fetch.Wired in the same way as the OpenAI backend:
Backendunion + config block, validation (validatePegasusModel), pipeline dispatch, thetranscribecommand, and the JSON schema. Docs updated in getting-started / commands / configuration.How it was tested
bun test— added no-network dry-run + validation tests (backend selection, model validation, URL-only guard); all pass. The 2 pre-existing "real WAV" failures are unrelated (they need a local whisper model installed).biome check— clean on all changed/new files.POST /v1.3/analyze(stream:false) with a real public MP4:transcribePegasusreturns text and writes.srt/.txt. A key-gated live test is included (TWELVELABS_API_KEY+TRX_TEST_VIDEO_URL), skipped when unset.You can grab a free API key at https://twelvelabs.io — there's a generous free tier.