feat: Add analyze_video tool using TwelveLabs Pegasus#35
Open
mohit-twelvelabs wants to merge 1 commit into
Open
feat: Add analyze_video tool using TwelveLabs Pegasus#35mohit-twelvelabs wants to merge 1 commit into
mohit-twelvelabs wants to merge 1 commit into
Conversation
Adds an optional analyze_video tool that runs prompt-based video understanding (summary / Q&A) via TwelveLabs Pegasus. Unlike get_transcript, it reasons over on-screen content, so it works for videos with little or no speech. The tool is fully opt-in: it only activates when called and requires a TWELVELABS_API_KEY. The existing transcript functionality is unchanged and needs no TwelveLabs configuration.
WalkthroughTwelveLabs Pegasus 기반 ChangesTwelveLabs 비디오 분석
Sequence Diagram(s)sequenceDiagram
participant "MCP Client" as MCPClient
participant "src/index.ts handleToolCall" as HandleToolCall
participant "src/twelvelabs-analyzer.ts analyzeVideo" as AnalyzeVideo
participant "TwelveLabs SDK" as TwelveLabsSDK
participant "TwelveLabs API" as TwelveLabsAPI
MCPClient->>HandleToolCall: analyze_video args
HandleToolCall->>AnalyzeVideo: url, prompt, model, max_tokens
AnalyzeVideo->>TwelveLabsSDK: create client and call analyze
TwelveLabsSDK->>TwelveLabsAPI: submit analysis request
TwelveLabsAPI-->>TwelveLabsSDK: response.data / finishReason
TwelveLabsSDK-->>AnalyzeVideo: analyze result
AnalyzeVideo-->>HandleToolCall: text, model, finishReason
HandleToolCall-->>MCPClient: content, structuredContent
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (1)
src/twelvelabs-analyzer.ts (1)
28-29: 🎯 Functional Correctness | 🔵 Trivial | 💤 Low value
maxTokens범위 검증 부재 (선택).주석에
2-4096 (pegasus1.2)로 명시되어 있으나 어댑터/호출부 어디에서도 범위를 검증하지 않아 잘못된 값은 API 호출 시점에야 거부됩니다. 사전 검증을 추가하면 더 명확한 오류 메시지를 줄 수 있습니다. pegasus1.5는 토큰 한도가 다르므로(모델별 컨텍스트 윈도) 모델별 검증이 바람직합니다.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/twelvelabs-analyzer.ts` around lines 28 - 29, The maxTokens option is only documented in the Twelvelabs analyzer but never validated before requests are sent. Add upfront range checking in the twelvelabs analyzer flow, ideally where the request is built or initialized in twelvelabs-analyzer, so invalid values fail early with a clear error. Make the validation model-aware (for example, pegasus1.2 versus pegasus1.5) and enforce the appropriate token window before calling the API.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@src/twelvelabs-analyzer.ts`:
- Around line 28-29: The maxTokens option is only documented in the Twelvelabs
analyzer but never validated before requests are sent. Add upfront range
checking in the twelvelabs analyzer flow, ideally where the request is built or
initialized in twelvelabs-analyzer, so invalid values fail early with a clear
error. Make the validation model-aware (for example, pegasus1.2 versus
pegasus1.5) and enforce the appropriate token window before calling the API.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 0f0a4263-73b2-41f9-921e-c3cd7c977d39
⛔ Files ignored due to path filters (1)
package-lock.jsonis excluded by!**/package-lock.json
📒 Files selected for processing (5)
README.mdpackage.jsonsrc/evals/evals.tssrc/index.tssrc/twelvelabs-analyzer.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi! I'm Mohit, I work at TwelveLabs (@mohit-twelvelabs).
What this adds
A new optional
analyze_videoMCP tool that runs prompt-based video understanding via TwelveLabs Pegasus. Whereget_transcriptonly captures spoken words, Pegasus reasons over what is actually shown on screen — so it produces useful summaries and answers even for videos with little or no speech (product demos, gameplay, b-roll, music videos).Inputs:
url(a direct video URL),prompt(instruction/question, defaults to a general summary),model(pegasus1.2/pegasus1.5), andmax_tokens. It returns bothcontenttext andstructuredContent, matching the shape of the existingget_transcripttool.Why it helps this project
This server is the go-to MCP tool for getting text out of videos. Pegasus extends that from "what was said" to "what happened on screen", which is a natural complement for the summarize/QA workflows people already point at this server — and it covers the large set of videos that have poor or no captions.
Opt-in / non-breaking
TWELVELABS_API_KEYenv var; without it, the existing transcript functionality is completely unchanged and needs zero TwelveLabs configuration.twelvelabs-jsSDK is added as the single new dependency.How it was tested
npm run buildpasses (strict TypeScript).pegasus1.2, a public sample.mp4) returns a correct on-screen summary, exercising the fullanalyze({ video: { type: "url" }, prompt }) -> datapath.analyze_videoentry to the existingmcp-evalssuite.You can grab a free API key at https://twelvelabs.io — there's a generous free tier.
Summary by CodeRabbit