fix: Vertex AI Express Mode — remove responseSchema+googleSearch conflict

[groupthinking]

Problem

Vertex AI does not support controlled generation (responseSchema) combined with the googleSearch tool, causing 400 errors:

Unable to submit request because controlled generation is not supported with Search tool

Changes

• gemini-client.ts: Prioritize Vertex_AI_API_KEY in resolution order, support GOOGLE_GENAI_USE_VERTEXAI env var, Express Mode docs link
• gemini-video-analyzer.ts: Remove responseSchema/responseMimeType, enforce JSON structure via prompt instructions
• extract-events/route.ts: Same fix for extractWithGemini() and inline Gemini search calls, remove dead geminiResponseSchema/Type import
• Strip markdown code fences from LLM responses before JSON parsing

Validation

Tested end-to-end with Vertex AI Express Mode key:

• Amy Cuddy TED Talk → 11 events, 20 transcript segments, full summary ✅
• Rick Astley music video → correct identification, 7 transcript segments ✅
• Server logs clean, no 400 errors ✅

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
v0-uvai	Building	Preview, Comment, Open in v0	Feb 28, 2026 10:15pm

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses a critical incompatibility in Vertex AI Express Mode where using responseSchema for controlled generation conflicted with the googleSearch tool, leading to API errors. The changes involve refactoring the Gemini client configurations to remove the problematic responseSchema and responseMimeType when googleSearch is active. Instead, JSON output is now enforced through explicit prompt instructions and a post-processing step to clean LLM responses, ensuring reliable structured data extraction. Additionally, the logic for resolving Gemini API keys and activating Vertex AI Express Mode has been refined for better clarity and prioritization.

Highlights

• Resolved Vertex AI Conflict: Eliminated the responseSchema and responseMimeType configuration when using the googleSearch tool in Vertex AI Express Mode, which previously caused 400 errors due to incompatibility.
• Improved JSON Parsing Robustness: Implemented a post-processing step to strip markdown code fences from LLM responses before JSON parsing, ensuring successful data extraction even if the model includes them.
• Enhanced Vertex AI Key Resolution: Prioritized Vertex_AI_API_KEY in the API key resolution order and introduced support for the GOOGLE_GENAI_USE_VERTEXAI environment variable for clearer Vertex AI Express Mode activation.
• Streamlined Vertex AI Configuration: Removed the need for explicit GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_LOCATION environment variables for Vertex AI Express Mode, simplifying setup.

Changelog

• apps/web/src/app/api/extract-events/route.ts
  • Removed the import of Type from @google/genai.
  • Deleted the geminiResponseSchema object.
  • Removed responseMimeType and responseSchema from Gemini API call configurations.
  • Added logic to strip markdown code fences from the LLM response before JSON parsing.
• apps/web/src/lib/gemini-client.ts
  • Reordered resolveGeminiApiKey to prioritize Vertex_AI_API_KEY.
  • Modified shouldUseVertexAI to check for GOOGLE_GENAI_USE_VERTEXAI environment variable and simplified logic.
  • Updated comments to reflect Vertex AI Express Mode details and removed references to GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_LOCATION in the GoogleGenAI constructor for Vertex AI mode.
• apps/web/src/lib/gemini-video-analyzer.ts
  • Removed the import of Type from @google/genai.
  • Deleted the responseSchema object.
  • Removed responseMimeType and responseSchema from Gemini API call configurations.
  • Added explicit JSON structure requirements to the system instruction prompt.
  • Added logic to strip markdown code fences from the LLM response before JSON parsing.

Activity

• No human activity has occurred on this pull request yet.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[sentry]

Bug: Setting GOOGLE_GENAI_USE_VERTEXAI=true without a Vertex_AI_API_KEY causes a runtime authentication failure because an incompatible GEMINI_API_KEY is used for the Vertex AI client.
_{Severity: MEDIUM}

Suggested Fix

Add validation in getGeminiClient() or shouldUseVertexAI() to ensure that when Vertex AI mode is selected, a Vertex_AI_API_KEY is present. Alternatively, document the key requirements for the GOOGLE_GENAI_USE_VERTEXAI flag.

Prompt for AI Agent

Review the code at the location below. A potential bug has been identified by an AI
agent.
Verify if this is a real issue. If it is, propose a fix; if not, explain why it's not
valid.

Location: apps/web/src/lib/gemini-client.ts#L31-L40

Potential issue: The code introduces a new environment variable
`GOOGLE_GENAI_USE_VERTEXAI` to select the Vertex AI client but lacks validation to
ensure a corresponding `Vertex_AI_API_KEY` is also provided. If a user sets
`GOOGLE_GENAI_USE_VERTEXAI` to 'true' while only a standard `GEMINI_API_KEY` is
configured, the application will incorrectly attempt to authenticate with Vertex AI
using an incompatible key. This misconfiguration leads to a runtime authentication
failure on the first API call, returning an unclear error message to the user.

_{Did we get this right? 👍 / 👎 to inform future reviews.}

[Copilot]

Pull request overview

Removes responseSchema/controlled generation when using the googleSearch tool to avoid Vertex AI Express Mode 400 errors, and instead relies on prompt-enforced JSON plus post-processing before parsing.

Changes:

• Drop responseSchema/responseMimeType from Gemini calls that use googleSearch and enforce JSON shape via system/user prompts.
• Add markdown code-fence stripping before JSON.parse.
• Adjust Gemini client auth selection to support Vertex AI Express Mode and an opt-in env var.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

File	Description
apps/web/src/lib/gemini-video-analyzer.ts	Removes controlled generation schema for search-tool requests; adds JSON-only prompting and response cleaning before parsing.
apps/web/src/lib/gemini-client.ts	Updates key resolution and switches Vertex AI usage to Express Mode (apiKey + vertexai), with an env override.
apps/web/src/app/api/extract-events/route.ts	Removes Gemini response schema usage for search-tool requests; adds response cleaning before JSON parsing.

[Copilot]

JSON.parse(cleaned)will throw whenresponse.textis empty/whitespace (cleaned becomes an empty string). Previously this function defaulted to'{}', so this is a behavior change that can crash callers on empty model responses; consider restoring a safe fallback (e.g., defaulting to '{}'`) or throwing a more descriptive error that includes the raw response for debugging.

Suggested change

	return JSON.parse(cleaned) as VideoAnalysisResult;
	const safeJson = cleaned || '{}';
	try {
	return JSON.parse(safeJson) as VideoAnalysisResult;
	} catch (error) {
	throw new Error(
	`Failed to parse Gemini response as JSON. Raw response:\n${resultText}`,
	);
	}

[Copilot]

The markdown-fence stripping logic is duplicated in multiple places (here and in the extract-events API route). Consider extracting a small shared helper (e.g., in a lib/util file) so future tweaks (like handling leading explanatory text, multiple code blocks, or alternate newline styles) stay consistent across the codebase.

[Copilot]

This documentation URL looks suspicious (Google Cloud docs are typically hosted under https://cloud.google.com/... rather than https://docs.cloud.google.com/...). If this link doesn’t resolve reliably, please replace it with the canonical Cloud docs URL to avoid broken references for future maintainers.

Suggested change

	* See: https://docs.cloud.google.com/vertex-ai/generative-ai/docs/start/express-mode/vertex-ai-express-mode-api-reference
	* See: https://cloud.google.com/vertex-ai/generative-ai/docs/start/express-mode/vertex-ai-express-mode-api-reference

[Copilot]

Env vars are commonly set to values like + "TRUE" + , + "1" + , or mixed-case via different tooling; the strict comparison to 'true' can silently fail. Consider normalizing (e.g., lowercasing) and/or accepting common truthy values to make configuration less brittle.

Suggested change

	if (process.env.GOOGLE_GENAI_USE_VERTEXAI === 'true') return true;
	const useVertexEnv = process.env.GOOGLE_GENAI_USE_VERTEXAI;
	if (useVertexEnv && ['true', '1'].includes(useVertexEnv.trim().toLowerCase())) return true;

[gemini-code-assist]

Code Review

This pull request correctly addresses the incompatibility between Vertex AI's controlled generation (responseSchema) and the googleSearch tool by removing the schema and enforcing JSON output via prompt engineering. However, this approach introduces a significant Prompt Injection vulnerability, as the application now relies on the LLM following instructions embedded in prompts that contain unsanitized user input, specifically transcript, videoTitle, and videoUrl. It is crucial to remediate this by sanitizing user input and using clear delimiters to separate user content from system instructions in LLM prompts. Additionally, the changes to gemini-client.ts to better support Vertex AI Express Mode are well-implemented. I've also provided specific comments in apps/web/src/app/api/extract-events/route.ts to improve code quality, address duplication, and enhance type safety.

[gemini-code-assist]

The application directly concatenates untrusted user input (transcript, videoTitle, and videoUrl) into the prompt sent to the Gemini LLM. An attacker can provide a specially crafted transcript or video title that includes instructions to the LLM, potentially causing it to ignore its original instructions, return malicious content, or abuse the enabled googleSearch tool. This is a classic Prompt Injection vulnerability.

[gemini-code-assist]

The analyzeVideoWithGemini function concatenates the videoUrl parameter directly into the system instruction and the user prompt. Since videoUrl is untrusted input, this allows for prompt injection attacks. An attacker could provide a URL followed by malicious instructions that the LLM might execute.

[gemini-code-assist]

The logic to strip markdown fences and parse the JSON response is duplicated here, again on lines 169-171, and also in apps/web/src/lib/gemini-video-analyzer.ts.

To improve maintainability and reduce code duplication, consider creating a shared utility function. This function could also help with adding proper types.

For example, you could create a utility in a shared lib file:

export function cleanAndParseJson<T>(text: string | undefined): T {
  const cleaned = (text ?? '').trim().replace(/^```(?:json)?\s*\n?/i, '').replace(/\n?```\s*$/i, '');
  // Return an empty object for empty/nullish text to avoid JSON.parse errors
  return JSON.parse(cleaned || '{}') as T;
}

You could then use this function in all three places, simplifying the code and ensuring consistency.