Enhance NotebookLM integration and frontend video ingestion pipeline

[groupthinking]

This pull request introduces significant improvements to the application's AI-driven video analysis pipeline, enhances transcript extraction strategies, and updates dependencies and configuration files to support new features. The main focus is on enabling robust, multi-strategy extraction of structured data from YouTube videos, with graceful fallback between OpenAI and Gemini providers, and providing a full end-to-end pipeline for video-to-software generation.

Key changes:

1. End-to-End Pipeline and Video Analysis Enhancements

• Added a new /api/pipeline endpoint implementing an end-to-end pipeline: YouTube URL → Video Analysis → Code Generation → Deployment → Live URL, with fallback between a backend FastAPI pipeline and Gemini-based analysis if the backend is unavailable.
• Enhanced /api/extract-events/route.ts to support both OpenAI and Gemini for structured event extraction, including a new Gemini schema using the @google/genai Type system and direct video analysis via Google Search when only a video URL is provided. [1] [2] [3]

2. Transcript Extraction Improvements

• Refactored /api/transcribe/route.ts to use a multi-strategy approach for transcript extraction: tries backend YouTube transcript API, then Gemini with Google Search, then OpenAI web search, and finally direct audio transcription, with YouTube metadata enrichment for better context. [1] [2] [3]

3. Dependency and Configuration Updates

• Added @google/genai as a new dependency in apps/web/package.json to enable advanced Gemini features.
• Added .firebase/.graphqlrc and .firebaserc configuration files for improved project setup and GraphQL schema management. [1] [2]
• Updated the Dockerfile to create new data directories for generated projects and YouTube-processed videos, improving data management for video analysis and deployment.

4. Developer Tooling

• Updated .vscode/settings.json to enable automatic outline generation for Gemini Code Assist, enhancing the developer experience.

---

These changes collectively improve the reliability, flexibility, and capabilities of the application's AI-powered video analysis and code generation workflows.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
v0-uvai	Ready	Preview, Comment, Open in v0	Mar 4, 2026 2:36am

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly upgrades the application's video analysis and software generation capabilities. It establishes a comprehensive end-to-end pipeline that transforms YouTube videos into deployed software, integrating advanced AI models like Gemini and OpenAI for intelligent content extraction and analysis. The changes focus on enhancing reliability through multi-provider strategies, improving transcript accuracy, and providing a more robust and observable workflow for video-driven development.

Highlights

• Enhanced AI-Driven Video Analysis Pipeline: Implemented a robust, multi-strategy approach for structured data extraction from YouTube videos, supporting both OpenAI and Gemini providers with graceful fallback mechanisms. This includes direct video analysis via Google Search when only a video URL is provided.
• End-to-End Video-to-Software Generation Pipeline: Introduced a new /api/pipeline endpoint that orchestrates a full workflow from YouTube URL to code generation, deployment, and a live URL. It features fallback between a backend FastAPI pipeline and Gemini-based analysis if the backend is unavailable.
• Improved Transcript Extraction Strategies: Refactored the /api/transcribe endpoint to employ a multi-strategy approach for transcript extraction, prioritizing backend YouTube API, then Gemini with Google Search, followed by OpenAI web search, and finally direct audio transcription, enriched with YouTube metadata.
• Gemini Integration and Configuration Updates: Added @google/genai as a new dependency and introduced a shared Gemini client factory (gemini-client.ts) to manage API keys and support both Gemini API and Vertex AI Express Mode. Configuration files for Firebase and Docker were updated to support new data directories and GraphQL schema management.
• Developer Experience and Tooling: Updated VS Code settings to enable automatic outline generation for Gemini Code Assist, enhancing the developer experience. Frontend components were also updated to display pipeline deployment results and trigger the new end-to-end pipeline.
• Client-Side Built-in AI Fallback: Introduced a new React hook (use-builtin-ai.ts) and related services to leverage Chrome Built-in AI capabilities (Prompt API and Summarizer API) for on-device text processing, providing a client-side fallback when server-side AI keys are unavailable.
• CloudEvents for Pipeline Observability: Added a new cloudevents.ts utility to publish standardized CloudEvents at each video processing stage, enabling better observability and integration with downstream consumers.

Changelog

• .firebase/.graphqlrc
  • Added new GraphQL configuration file for Firebase Data Connect.
• .firebaserc
  • Added Firebase project configuration file.
• .gitignore
  • Added '.next/' to ignore Next.js build output.
• .vscode/settings.json
  • Enabled automatic outline generation for Gemini Code Assist.
• Dockerfile
  • Created new data directories for generated projects, YouTube processed videos, and temporary UVAI data.
• apps/web/package.json
  • Added @google/genai dependency for advanced Gemini features.
• apps/web/src/app/api/extract-events/route.ts
  • Removed direct GoogleGenerativeAI import and getGemini function.
  • Imported Type from @google/genai and shared Gemini client utilities.
  • Defined a new geminiResponseSchema using the @google/genai Type system.
  • Updated extractWithGemini to use the new shared client and gemini-3-pro-preview model with responseSchema and googleSearch tool.
  • Modified POST handler to accept videoUrl for direct Gemini analysis and implemented fallback logic for OpenAI quota issues.
  • Updated error messages to reflect multi-provider configuration.
• apps/web/src/app/api/pipeline/route.ts
  • Added new API endpoint for an end-to-end pipeline: YouTube URL to deployed software.
  • Implemented strategies for full backend pipeline (FastAPI) and Gemini-only analysis fallback.
  • Integrated CloudEvents publishing for pipeline stages.
• apps/web/src/app/api/transcribe/route.ts
  • Refactored OpenAI client initialization.
  • Imported YouTube metadata utilities and shared Gemini client utilities.
  • Updated backend URL validation to check for availability.
  • Implemented a multi-strategy approach for transcript extraction: backend YouTube API, Gemini with Google Search, OpenAI web search, and direct audio STT.
  • Added YouTube metadata enrichment for Gemini and OpenAI web search strategies.
  • Included logic to filter out generic AI responses for web search results.
  • Updated error messages for API key configuration.
• apps/web/src/app/api/video/route.ts
  • Imported CloudEvents, Gemini video analyzer, and Gemini client utilities.
  • Added getBaseUrl helper function.
  • Refactored POST handler to support multiple analysis strategies: full backend, Gemini agentic analysis, and frontend-only transcribe/extract chain.
  • Integrated CloudEvents publishing for various video analysis stages.
  • Updated GET endpoint to reflect backend availability and frontend-only mode.
• apps/web/src/app/dashboard/page.tsx
  • Imported PipelineResult type from the dashboard store.
  • Updated VideoCard component to conditionally display pipeline deployment results (live URL, GitHub repo, code generation details).
  • Adjusted processing stage indicators based on whether a pipeline result is present.
  • Added a new 'Deploy' button and integrated handleDeployPipeline callback.
• apps/web/src/app/layout.tsx
  • Removed next/font/google imports to prevent build failures in sandboxed environments.
  • Replaced font imports with direct Google Fonts <link> in the HTML head.
• apps/web/src/app/page.tsx
  • Updated example YouTube URLs for demonstration purposes.
• apps/web/src/hooks/use-builtin-ai.ts
  • Added new React hook useBuiltInAI to expose Chrome Built-in AI capabilities for client-side summarization and event extraction.
• apps/web/src/lib/cloudevents.ts
  • Added new utility to define and publish CloudEvents for pipeline observability, supporting webhook delivery or local logging.
• apps/web/src/lib/gemini-client.ts
  • Added new utility to provide a shared Gemini/Google AI client, supporting both Gemini API and Vertex AI Express Mode authentication.
• apps/web/src/lib/gemini-video-analyzer.ts
  • Added new agentic video intelligence engine using Gemini and Google Search grounding.
  • Defined VideoAnalysisResult interface and a detailed responseSchema for structured output.
  • Implemented buildSystemInstruction for Gemini model to guide agentic analysis.
• apps/web/src/lib/services/agent-service.ts
  • Added sendA2AMessage for inter-agent communication.
  • Added getA2ALog to retrieve agent-to-agent message logs.
• apps/web/src/lib/services/builtin-ai.ts
  • Added new service for Chrome Built-in AI, including type declarations and functions for checking capabilities, summarizing transcripts, and extracting events.
• apps/web/src/lib/youtube-metadata.ts
  • Added new utility to fetch YouTube video metadata (title, description, chapters) by scraping the watch page without an API key.
  • Included functions to extract video IDs, parse chapters, and format metadata as AI context.
• apps/web/src/store/dashboard-store.ts
  • Defined PipelineResult interface to store outcomes of the end-to-end pipeline.
  • Updated Video interface to include pipelineResult.
  • Added deployPipeline action to handle the end-to-end deployment workflow, including progress updates and activity logging.
• dataconnect/.dataconnect/pgliteData/pg17/PG_VERSION
  • Added PostgreSQL version file for Data Connect.

Activity

• Initial implementation of the end-to-end video-to-software pipeline.
• Refactored video analysis and transcript extraction logic to support multiple AI providers and fallback strategies.
• Integrated new Gemini client and agentic analysis capabilities.
• Updated frontend to reflect new pipeline features and display deployment results.
• Added client-side AI fallback using Chrome Built-in AI.
• Introduced CloudEvents for improved pipeline observability.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[Copilot]

Pull request overview

Improves the frontend/serverless video ingestion pipeline with multi-strategy transcript/event extraction, adds an end-to-end “video → deploy” pipeline endpoint, and introduces Gemini/Chrome Built-in AI integrations plus supporting config/deps.

Changes:

• Added /api/pipeline endpoint and frontend store/UI support for end-to-end deployment results
• Implemented Gemini-based agentic video analysis (googleSearch grounding) and multi-strategy transcript/extraction fallbacks
• Added CloudEvents publishing, Gemini client utilities, and updated dependencies/configuration

Reviewed changes

Copilot reviewed 21 out of 12310 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
dataconnect/.dataconnect/pgliteData/pg17/PG_VERSION	Adds a local pglite Postgres version file under dataconnect state
apps/web/src/store/dashboard-store.ts	Adds pipeline result types and a deployPipeline workflow in the dashboard store
apps/web/src/lib/youtube-metadata.ts	Adds YouTube metadata scraping + chapter parsing utilities
apps/web/src/lib/services/builtin-ai.ts	Adds Chrome Built-in AI client-side fallback helpers
apps/web/src/lib/services/agent-service.ts	Adds A2A send/log API calls to the agent service
apps/web/src/lib/gemini-video-analyzer.ts	Adds Gemini agentic video analyzer with responseSchema + googleSearch
apps/web/src/lib/gemini-client.ts	Adds shared Gemini client factory with key/mode resolution
apps/web/src/lib/cloudevents.ts	Adds CloudEvents event creation + publishing helper
apps/web/src/hooks/use-builtin-ai.ts	Adds React hook for Built-in AI capability checks and helpers
apps/web/src/app/page.tsx	Updates example YouTube URLs shown on the homepage
apps/web/src/app/layout.tsx	Switches from next/font to Google Fonts to avoid build failures
apps/web/src/app/dashboard/page.tsx	Updates dashboard UI to support pipeline deployments + show pipeline results
apps/web/src/app/api/video/route.ts	Adds multi-strategy backend/Gemini/frontend-chain analysis with CloudEvents
apps/web/src/app/api/transcribe/route.ts	Refactors transcription into multi-strategy backend/Gemini/OpenAI/STT flow with metadata
apps/web/src/app/api/pipeline/route.ts	Adds end-to-end pipeline endpoint with backend-first + Gemini analysis fallback
apps/web/src/app/api/extract-events/route.ts	Adds Gemini structured extraction via @google/genai schema + direct videoUrl extraction
apps/web/package.json	Adds @google/genai dependency
Dockerfile	Creates/chowns additional data directories used by the pipeline
.vscode/settings.json	Enables Gemini Code Assist automatic outline generation
.firebaserc	Adds Firebase project configuration
.firebase/.graphqlrc	Adds GraphQL schema/document configuration for Data Connect

[Copilot]

This file appears to be generated local state for pglite/dataconnect (embedded DB data directory). Committing it will likely create noisy diffs and environment-specific state in the repo. Consider removing it from version control and adding the broader .dataconnect/pgliteData/ path to .gitignore (or storing DB state outside the repo).

Suggested change

	17
	# Placeholder file; do not commit pglite/dataconnect DB state.
	# The real PG_VERSION file should be generated locally by the embedded DB.

[Copilot]

The watch URL regex only matches youtube.com/watch?v=... when v is the first query param. Common URLs like .../watch?si=...&v=VIDEOID or .../watch?time_continue=1&v=VIDEOID won’t match, causing metadata fetch to fail for valid YouTube links. Consider broadening the pattern to allow other params in any order (e.g., match [?&]v=).

[Copilot]

The module docstring claims events are appended to /tmp/cloudevents.jsonl when no backend is configured, but publishEvent() currently only logs to the console when CLOUDEVENTS_WEBHOOK_URL is unset. Either implement the JSONL append (server-side only), or update the documentation to match the current behavior.

[Copilot]

The module docstring claims events are appended to /tmp/cloudevents.jsonl when no backend is configured, but publishEvent() currently only logs to the console when CLOUDEVENTS_WEBHOOK_URL is unset. Either implement the JSONL append (server-side only), or update the documentation to match the current behavior.

[Copilot]

When the pipeline returns an OK response but indicates a non-success status, the video is marked failed but still shows progress: 100 and a Deployed: title. This produces misleading UI for failed runs. Consider setting the title/progress based on the computed status (e.g., only use Deployed:/100% when status is complete, otherwise use a failure title and a non-100 progress).

Suggested change

	updateVideo(id, {
	status: result.status === 'success' || result.status === 'complete' ? 'complete' : 'failed',
	progress: 100,
	title: `Deployed: ${url.length > 40 ? url.substring(0, 37) + '…' : url}`,
	const isSuccess = result.status === 'success' || result.status === 'complete';
	const computedStatus = isSuccess ? 'complete' : 'failed';
	const progress = isSuccess ? 100 : 90;
	const truncatedUrl = url.length > 40 ? url.substring(0, 37) + '…' : url;
	const titlePrefix = isSuccess ? 'Deployed' : 'Deployment failed';
	updateVideo(id, {
	status: computedStatus,
	progress,
	title: `${titlePrefix}: ${truncatedUrl}`,

[Copilot]

The documented strategy order doesn’t match the implementation: the code attempts Gemini w/ googleSearch before OpenAI web_search. Also the final strategy uses gpt-4o-mini-transcribe (not Whisper). Updating the comment to reflect the actual order/tools would prevent confusion for future maintainers.

Suggested change

	* 2. OpenAI Responses API with web_search (finds transcripts online)
	* 3. Gemini fallback (if OpenAI unavailable)
	* 4. Direct audio STT via OpenAI Whisper
	* 2. Gemini with Google Search grounding (primary for YouTube transcripts)
	* 3. OpenAI Responses API with web_search (fallback when Gemini or Google Search are unavailable)
	* 4. Direct audio STT via OpenAI gpt-4o-mini-transcribe

[Copilot]

This error message is misleading because extraction can succeed with OPENAI_API_KEY alone, and failure here may also reflect provider/runtime errors (not only missing Gemini). Consider revising the message to mention both keys (OPENAI_API_KEY/GEMINI_API_KEY) and/or distinguish between “no provider configured” vs “all providers failed” based on the environment and attempted paths.

Suggested change

	error: 'No AI API key configured or all extraction attempts failed. Set GEMINI_API_KEY.',
	error:
	'No AI providers succeeded. Either no API keys are configured (missing OPENAI_API_KEY and/or GEMINI_API_KEY) or all extraction attempts failed at runtime.',

[gemini-code-assist]

Code Review

This pull request introduces a significant set of features, enhancing the video analysis pipeline with multi-provider support (OpenAI and Gemini) and robust fallback strategies, establishing a new end-to-end pipeline from video to deployed software. However, it introduces critical security concerns, including a Server-Side Request Forgery (SSRF) vulnerability in the transcription API and multiple Prompt Injection points in the AI-driven analysis logic. Additionally, a critical issue with unsafe JSON parsing that could lead to crashes, a high-severity bug due to a typo in a model name, and several medium-severity suggestions for improving ID generation and type safety have been identified. These issues should be addressed to ensure the application's robustness and security.

[gemini-code-assist]

This direct call to JSON.parse() is unsafe. If response.text from the Gemini API is an empty string (which is possible if the model returns no content), this will throw an unhandled exception and cause the API route to crash with a 500 error. You should gracefully handle the case of an empty or invalid JSON string before parsing.

Suggested change

	const text = response.text ?? '';
	return JSON.parse(text);
	const text = response.text ?? '';
	return text ? JSON.parse(text) : {};

[gemini-code-assist]

The JSON.parse() call is unsafe. If the resultText from the AI is an empty string or not valid JSON, this will throw an unhandled exception. Given that this is a core analysis function, it should be more robust. Please wrap the parsing in a try...catch block to handle potential errors gracefully.

  try {
    return JSON.parse(resultText) as VideoAnalysisResult;
  } catch (e) {
    console.error('Failed to parse VideoAnalysisResult from Gemini:', resultText, e);
    throw new Error('Failed to parse JSON response from Gemini for video analysis.');
  }

[gemini-code-assist]

The audioUrl parameter is taken directly from the user request and used in a fetch call without any validation or sanitization. This allows an attacker to make the server perform arbitrary network requests, potentially reaching internal services or metadata endpoints (SSRF).

Suggested change

	const audioResponse = await fetch(audioUrl);
	// Validate audioUrl before fetching
	try {
	const parsedUrl = new URL(audioUrl);
	if (!['http:', 'https:'].includes(parsedUrl.protocol)) {
	throw new Error('Invalid protocol');
	}
	// Add additional checks for internal IP ranges if necessary
	} catch (e) {
	return NextResponse.json({ error: 'Invalid audioUrl' }, { status: 400 });
	}
	const audioResponse = await fetch(audioUrl);

[gemini-code-assist]

The model name gemini-3-pro-preview appears to be a typo and is not a valid Google Gemini model name. This will cause all API calls using this function to fail. You likely intended to use a model from the Gemini 1.5 family, such as gemini-1.5-pro-latest or a specific preview version. Please correct the model name to a valid one.

    model: 'gemini-1.5-pro-latest',

[gemini-code-assist]

The videoUrl parameter is directly embedded into the systemInstruction for the Gemini model. An attacker can provide a malicious URL containing instructions to override the system prompt and manipulate the LLM's behavior (Prompt Injection). This is particularly dangerous as the googleSearch tool is enabled, which could be abused to perform arbitrary searches.

  const videoIdMatch = videoUrl.match(/[?&]v=([^&]+)/);
  const videoId = videoIdMatch ? videoIdMatch[1] : 'Unknown';
  if (!videoIdMatch) {
    // Handle invalid URL or log warning
  }

[gemini-code-assist]

Using Date.now().toString(36) to generate an ID is not guaranteed to be unique, especially if multiple requests are processed concurrently. This could lead to collisions and unexpected behavior. It's better to use a cryptographically secure random UUID to ensure uniqueness.

Suggested change

	id: `pipeline_${Date.now().toString(36)}`,
	id: `pipeline_${crypto.randomUUID()}`,

[gemini-code-assist]

Using Date.now().toString(36) to generate an ID is not guaranteed to be unique, especially if multiple requests are processed concurrently. This could lead to collisions and unexpected behavior. It's better to use a cryptographically secure random UUID to ensure uniqueness.

Suggested change

	id: `pipeline_${Date.now().toString(36)}`,
	id: `pipeline_${crypto.randomUUID()}`,

[gemini-code-assist]

The type for the extraction variable is defined inline. According to the repository style guide, functions should have strict type hinting, which is better achieved with clear, reusable, and named types. Defining complex types inline can reduce readability and reusability.

Consider extracting this into a named interface or type alias, for example:

interface ExtractionResult {
  events?: Array<{ type: string; title: string; description?: string; timestamp?: string; priority?: string }>;
  actions?: Array<{ title: string }>;
  summary?: string;
  topics?: string[];
}

// ... later in the function
let extraction: ExtractionResult = {};

References

1. All functions must have strict type hinting. Using a named type alias or interface for complex objects improves clarity and reusability, which aligns with the principle of strict typing. ^(link)

[gemini-code-assist]

Using Date.now().toString() for the id is not guaranteed to be unique. This can lead to issues in React when this ID is used as a key for list items and for state updates, especially if multiple actions are dispatched in quick succession. It's safer to use crypto.randomUUID() to ensure uniqueness.

Suggested change

	const id = Date.now().toString();
	const id = crypto.randomUUID();