Add frontend-only video analysis pipeline with multi-strategy transcription#33
Conversation
…ment The core pipeline previously required the Python backend to be running. When deployed to Vercel (https://v0-uvai.vercel.app/), the backend is unavailable, causing all video analysis to fail immediately. Changes: - /api/video: Falls back to frontend-only pipeline (transcribe + extract) when the Python backend is unreachable, with 15s timeout - /api/transcribe: Adds Gemini fallback when OpenAI is unavailable, plus 8s timeout on backend probe to avoid hanging on Vercel - layout.tsx: Loads Google Fonts via <link> instead of next/font/google to avoid build failures in offline/sandboxed CI environments - page.tsx: Replace example URLs with technical content (3Blue1Brown neural networks, Karpathy LLM intro) instead of rick roll / zoo videos - gemini_service.py: Gate Vertex AI import behind GOOGLE_CLOUD_PROJECT env var to prevent 30s+ hangs on the GCE metadata probe - agent_gap_analyzer.py: Fix f-string backslash syntax errors (Python 3.11) https://claude.ai/code/session_015Pd3a6hinTenCNrPRGiZqE
This comment was marked as spam.
This comment was marked as spam.
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly enhances the robustness and flexibility of the video analysis system by introducing a resilient dual-strategy architecture for video processing and a comprehensive multi-provider, multi-strategy approach for transcription. These changes ensure that the application can continue to function effectively even when the full Python backend is inaccessible, common in serverless deployment environments like Vercel, while also improving the reliability and quality of transcript generation. Highlights
Changelog
Activity
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
|
🚅 Deployed to the EventRelay-pr-33 environment in EventRelay
|
railway-app
bot
temporarily deployed
to
EventRelay / EventRelay-pr-33
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull request overview
This PR adds a resilient “dual-path” video analysis flow in the Next.js app: it attempts to use the FastAPI backend when reachable, and otherwise falls back to a frontend-only serverless chain for transcription + event extraction. It also expands /api/transcribe into a multi-provider, multi-strategy cascade and makes the Python Gemini service avoid slow Vertex AI imports unless explicitly enabled.
Changes:
- Implemented backend-first, frontend-fallback behavior for
/api/video, normalizing responses across strategies. - Added a multi-strategy transcription cascade in
/api/transcribe(backend YouTube captions → OpenAI web_search → Gemini → OpenAI STT). - Avoided slow Vertex AI SDK import at Python startup unless enabled via environment variables; adjusted font loading to remove
next/font/google.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| src/youtube_extension/services/ai/gemini_service.py | Makes Vertex AI import lazy to avoid slow metadata probes outside GCP. |
| src/youtube_extension/services/agents/agent_gap_analyzer.py | Tweaks markdown generation string joining for compatibility. |
| apps/web/src/app/page.tsx | Updates example YouTube URLs on the homepage. |
| apps/web/src/app/layout.tsx | Replaces next/font/google with <link>-based Google Fonts loading. |
| apps/web/src/app/api/video/route.ts | Adds backend-first analysis with a frontend-only fallback pipeline. |
| apps/web/src/app/api/transcribe/route.ts | Implements multi-provider / multi-strategy transcription cascade. |
| insights, | ||
| transcript_segments: result.transcript?.length || 0, | ||
| agents_used: result.orchestration_meta?.agents_used || [], | ||
| errors: result.errors || [], |
There was a problem hiding this comment.
transcript_segments is computed as result.transcript?.length, but the backend returns transcript as an object (e.g., { text, source, segments }), so this will always be 0/undefined and misreport segment counts. Use result.transcript?.segments?.length (or handle both array/object shapes) to keep the normalized response accurate.
| const controller = new AbortController(); | ||
| const timeout = setTimeout(() => controller.abort(), 8_000); | ||
|
|
||
| const ytResponse = await fetch(`${BACKEND_URL}/api/v1/transcript-action`, { | ||
| method: 'POST', | ||
| headers: { 'Content-Type': 'application/json' }, | ||
| body: JSON.stringify({ video_url: url, language }), | ||
| signal: controller.signal, | ||
| }); | ||
| clearTimeout(timeout); | ||
|
|
There was a problem hiding this comment.
The backend transcript fetch uses an AbortController timeout, but clearTimeout(timeout) only runs after a successful fetch. If the request aborts or throws, the timeout isn’t cleared. Wrap the fetch in try { ... } finally { clearTimeout(timeout) } to avoid leaving timers pending in the error path.
| # Vertex AI SDK probes the GCE metadata server on import which can hang | ||
| # for 5+ seconds outside GCP. Only import when explicitly requested via | ||
| # environment variables to keep startup fast in local / CI environments. | ||
| if os.getenv("GOOGLE_CLOUD_PROJECT") or os.getenv("ENABLE_VERTEX_AI"): | ||
| import vertexai | ||
| from vertexai.generative_models import GenerativeModel, Part | ||
|
|
||
| VERTEX_AVAILABLE = True | ||
| else: | ||
| VERTEX_AVAILABLE = False |
There was a problem hiding this comment.
ENABLE_VERTEX_AI gating uses a raw os.getenv(...) truthiness check. This will enable the Vertex AI import even when the variable is set to a falsey string like "0", defeating the intent to avoid slow metadata probing. Consider parsing ENABLE_VERTEX_AI as a boolean flag (e.g., accept only 1/true/yes) and/or prioritizing GOOGLE_CLOUD_PROJECT alone for auto-enable.
apps/web/src/app/api/video/route.ts
Outdated
| const controller = new AbortController(); | ||
| const timeout = setTimeout(() => controller.abort(), 15_000); | ||
|
|
||
| const response = await fetch(`${BACKEND_URL}/api/v1/transcript-action`, { | ||
| method: 'POST', | ||
| headers: { 'Content-Type': 'application/json' }, | ||
| body: JSON.stringify({ video_url: url, language: 'en' }), | ||
| signal: controller.signal, | ||
| }); | ||
| clearTimeout(timeout); |
There was a problem hiding this comment.
The backend fetch timeout isn’t cleared if fetch throws (e.g., due to abort/network error). Because clearTimeout(timeout) is after the await fetch, the timer can remain scheduled in the error path. Move clearTimeout into a finally block so it always runs.
![vercel[bot]](https://avatars.githubusercontent.com/in/8329?s=60&v=4){kind=small}
| } | ||
|
|
||
| let _gemini: GoogleGenerativeAI | null = null; | ||
| function getGemini() { |
There was a problem hiding this comment.
Gemini client initialization with empty string fallback silently creates invalid client when API key is missing
apps/web/src/app/api/video/route.ts
Outdated
|
|
||
| const origin = request.headers.get('x-forwarded-proto') | ||
| ? `${request.headers.get('x-forwarded-proto')}://${request.headers.get('host')}` | ||
| : new URL(request.url).origin; |
There was a problem hiding this comment.
Missing HTTP response status checking on fetch requests to /api/transcribe and /api/extract-events endpoints
…orgery Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
railway-app
bot
temporarily deployed
to
EventRelay / EventRelay-pr-33
|
@copilot open a new pull request to apply changes based on this feedback |
|
@groupthinking I've opened a new pull request, #34, to work on those changes. Once the pull request is ready, I'll request review from you. |
…orgery Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
railway-app
bot
temporarily deployed
to
EventRelay / EventRelay-pr-33
|
@copilot open a new pull request to apply changes based on the comments in this thread |
|
@groupthinking I've opened a new pull request, #35, to work on those changes. Once the pull request is ready, I'll request review from you. |
![sentry[bot]](https://avatars.githubusercontent.com/in/12637?s=60&v=4){kind=small}
apps/web/src/app/api/video/route.ts
Outdated
| let transcript = ''; | ||
| let transcript = ''; |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a significant improvement in resilience by adding a frontend-only fallback pipeline for video analysis and multi-provider support for transcription. This makes the system more robust, especially in environments where the Python backend might not be available. The lazy loading of vertexai in the Python backend is a thoughtful performance optimization. The code changes are generally well-structured. I've made a couple of suggestions to improve code readability and maintainability in the TypeScript API routes, specifically around prompt construction and complex data parsing logic. Overall, this is a solid set of changes that enhances the application's reliability and performance.
| let summaryText = 'Video analyzed successfully'; | ||
| const rawSummary = transcriptAction.summary; | ||
| if (typeof rawSummary === 'string') { | ||
| summaryText = rawSummary; | ||
| } else if (rawSummary && typeof rawSummary === 'object') { | ||
| summaryText = | ||
| rawSummary.content || | ||
| rawSummary.executive_summary || | ||
| (typeof rawSummary.raw === 'string' | ||
| ? (() => { | ||
| try { | ||
| const parsed = JSON.parse(rawSummary.raw.replace(/```json\n?|```/g, '')); | ||
| return parsed.executive_summary || parsed.summary || rawSummary.raw.slice(0, 200); | ||
| } catch { | ||
| return rawSummary.raw.slice(0, 200); | ||
| } | ||
| })() | ||
| : JSON.stringify(rawSummary).slice(0, 200)); | ||
| } |
There was a problem hiding this comment.
This logic to extract the summary text is quite complex and hard to follow due to the nested ternaries and an Immediately Invoked Function Expression (IIFE). To improve readability and maintainability, consider refactoring this block to be more linear and easier to understand. This will also make it easier to debug if the backend response format changes again.
let summaryText: string;
const rawSummary = transcriptAction.summary;
if (typeof rawSummary === 'string') {
summaryText = rawSummary;
} else if (rawSummary && typeof rawSummary === 'object') {
summaryText = rawSummary.content || rawSummary.executive_summary;
if (!summaryText && typeof rawSummary.raw === 'string') {
try {
const parsed = JSON.parse(rawSummary.raw.replace(/```json\n?|```/g, ''));
summaryText = parsed.executive_summary || parsed.summary || rawSummary.raw.slice(0, 200);
} catch {
summaryText = rawSummary.raw.slice(0, 200);
}
}
if (!summaryText) {
summaryText = JSON.stringify(rawSummary).slice(0, 200);
}
} else {
summaryText = 'Video analyzed successfully';
}Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>
railway-app
bot
temporarily deployed
to
EventRelay / EventRelay-pr-33
Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>
railway-app
bot
temporarily deployed
to
EventRelay / EventRelay-pr-33
Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>
railway-app
bot
temporarily deployed
to
EventRelay / EventRelay-pr-33
|
@copilot open a new pull request to apply changes based on this feedback |
|
@groupthinking I've opened a new pull request, #36, to work on those changes. Once the pull request is ready, I'll request review from you. |
|
@copilot apply changes based on the comments in this thread |
|
@groupthinking I've opened a new pull request, #37, to work on those changes. Once the pull request is ready, I'll request review from you. |
apps/web/src/app/api/video/route.ts
Outdated
| const transcribeRes = await fetch('/api/transcribe', { | ||
| method: 'POST', | ||
| headers: { 'Content-Type': 'application/json' }, | ||
| body: JSON.stringify({ url }), | ||
| }); |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
…h abort/error Co-authored-by: groupthinking <154503486+groupthinking@users.noreply.github.com>
…tch('/api/transcribe') and fetch('/api/extract-events') use relative URLs which don't resolve correctly in server-side Next.js code on production deployments like Vercel.
This commit fixes the issue reported at apps/web/src/app/api/video/route.ts:101
## Bug Analysis
**Why it happens:**
In Next.js API routes running on the server (Node.js runtime), the `fetch()` API requires absolute URLs. Unlike browsers which have an implicit base URL (the current origin), server-side code has no context for resolving relative URLs like `/api/transcribe`. The Node.js fetch implementation will fail to resolve these relative paths, resulting in TypeError or connection errors.
**When it manifests:**
- **Development (localhost:3000)**: Works accidentally because the request URL contains the host
- **Production (Vercel)**: Fails because the relative URL cannot be resolved to a valid absolute URL without proper host context
**What impact it has:**
The frontend-only pipeline fallback (Strategy 2) in lines 101-132 is completely broken in production. When the backend is unavailable (common on Vercel), the code attempts to use `/api/transcribe` and `/api/extract-events` serverless functions but fails due to unresolvable relative URLs. This causes the entire video analysis endpoint to fail when the backend is unavailable.
## Fix Explanation
**Changes made:**
1. Added a `getBaseUrl(request: Request)` helper function that extracts the absolute base URL from the incoming request object using `new URL(request.url)`
2. Updated line 108: `fetch('/api/transcribe', ...)` → `fetch(`${baseUrl}/api/transcribe`, ...)`
3. Updated line 127: `fetch('/api/extract-events', ...)` → `fetch(`${baseUrl}/api/extract-events`, ...)`
**Why it solves the issue:**
- The incoming `request` object contains the full URL including protocol and host
- By constructing an absolute URL from the request, we ensure the fetch calls work in both development and production
- This approach is more reliable than environment variables because it uses the actual request context, handling reverse proxies and different deployment configurations correctly
Co-authored-by: Vercel <vercel[bot]@users.noreply.github.com>
Co-authored-by: groupthinking <garveyht@gmail.com>
railway-app
bot
temporarily deployed
to
EventRelay / EventRelay-pr-33
groupthinking
left a comment
groupthinking
left a comment
There was a problem hiding this comment.
@copilot code review[agent] @claude[agent]
|
@groupthinking I've opened a new pull request, #38, to work on those changes. Once the pull request is ready, I'll request review from you. |
Fix AbortController timer leaks in fetch timeout paths
railway-app
bot
temporarily deployed
to
EventRelay / EventRelay-pr-33
railway-app
bot
temporarily deployed
to
EventRelay / EventRelay-pr-33
Summary
This PR implements a resilient, multi-strategy video analysis system that works both with and without a Python backend. The main
/api/videoendpoint now falls back to a frontend-only pipeline when the backend is unavailable (common on Vercel), and the/api/transcribeendpoint supports multiple AI providers and transcription strategies.Key Changes
Video Analysis API (
/api/video)/api/transcribe→/api/extract-eventsdirectly when backend unavailableTranscription API (
/api/transcribe)Python Backend (
gemini_service.py)GOOGLE_CLOUD_PROJECTorENABLE_VERTEX_AI) to avoid 5+ second metadata server probes in non-GCP environmentsLayout & Styling (
layout.tsx)<link>tags to avoid hard failures when Google Fonts API is unreachable (common in sandboxed CI/offline environments)Minor Updates
agent_gap_analyzer.pyto usechr(10)instead of'\n'for better compatibilityfrontend_pipeline: 'active'when backend unavailableImplementation Details
x-forwarded-protoandhostheaders for correct internal API routing on Verceltranscript_sourcefield to indicate which strategy succeededhttps://claude.ai/code/session_015Pd3a6hinTenCNrPRGiZqE