fix: restore full PK=998 pattern — responseSchema + googleSearch + gemini-3-pro-preview

[groupthinking]

What was wrong

PR #48 was a shortcut. When responseSchema + googleSearch failed with 400 on Vertex AI, I removed responseSchema entirely — destroying the structured output that PK=998 note specifically designed.

The actual error: gemini-2.5-flash and gemini-2.0-flash do NOT support responseSchema + googleSearch together on Vertex AI. But gemini-3-pro-preview does.

What this PR does

Restores the exact PK=998 pattern:

• responseMimeType: 'application/json' — guaranteed JSON
• responseSchema with Type system — enforced structure
• tools: [{ googleSearch: {} }] — search grounding
• model: 'gemini-3-pro-preview' — supports all three together
• e22Snippets field — production code snippets for E22 cloud solutions

Files changed

• gemini-video-analyzer.ts — full PK=998 schema restored
• extract-events/route.ts — restored geminiResponseSchema, Type, responseMimeType
• transcribe/route.ts — model upgrade

Validation

Tested with Vertex AI Express Mode key:

• Amy Cuddy TED Talk → 4 events with codeMapping+cloudService, 6 transcript segments, 4 actions, architecture+ingest ✅
• Trade School Grant video → 4 events, 6 segments, 4 actions, 7 topics ✅
• All responses are valid JSON via responseSchema enforcement ✅

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
v0-uvai	Building	Preview, Comment, Open in v0	Feb 28, 2026 10:37pm

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses a previous workaround by fully implementing the intended structured output pattern for Gemini-powered video analysis. By upgrading to the gemini-3-pro-preview model, the system can now simultaneously leverage responseSchema for guaranteed JSON output and googleSearch for enhanced grounding, leading to more reliable and richly structured data extraction from video content.

Highlights

• Model Upgrade: The Gemini model used for video analysis and transcription has been upgraded from gemini-2.0-flash or gemini-2.5-flash to gemini-3-pro-preview across relevant API routes and utility functions.
• Full PK=998 Pattern Restoration: The pull request restores the complete PK=998 pattern for structured output, which includes enforcing responseMimeType: 'application/json', utilizing responseSchema with the @google/genai Type system, and integrating tools: [{ googleSearch: {} }].
• Structured Output Enforcement: Explicit responseSchema definitions have been added to the Gemini model configurations, ensuring that the model's output strictly adheres to predefined JSON structures for events, actions, summaries, topics, and new e22Snippets.
• Simplified JSON Parsing: With the responseMimeType and responseSchema now enforcing JSON output directly from the model, the previous logic for stripping markdown code fences (```json) from the response text before parsing has been removed.
• Enhanced Video Analysis Output: The VideoAnalysisResult interface and the corresponding responseSchema in gemini-video-analyzer.ts have been extended to include e22Snippets, providing production-ready code snippets for E22 cloud solutions.

Changelog

• apps/web/src/app/api/extract-events/route.ts
  • Imported Type from @google/genai for schema definition.
  • Defined geminiResponseSchema to enforce structured JSON output for event extraction.
  • Updated the Gemini model to gemini-3-pro-preview for both direct and search-based extraction.
  • Configured responseMimeType to application/json and applied geminiResponseSchema to model calls.
  • Removed manual markdown fence stripping, relying on the model's enforced JSON output.
• apps/web/src/app/api/transcribe/route.ts
  • Upgraded the Gemini model to gemini-3-pro-preview for transcription tasks.
• apps/web/src/lib/gemini-video-analyzer.ts
  • Updated comments to reflect gemini-3-pro-preview's capability to combine responseSchema and googleSearch.
  • Imported Type from @google/genai.
  • Extended the VideoAnalysisResult interface to include e22Snippets.
  • Defined a comprehensive responseSchema using the Type system, incorporating all required fields including e22Snippets.
  • Modified buildSystemInstruction to include videoId and updated data structure requirements for the model.
  • Updated the Gemini model to gemini-3-pro-preview for video analysis.
  • Applied responseMimeType: 'application/json' and responseSchema to the model configuration.
  • Simplified JSON parsing by removing code to strip markdown fences.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request correctly restores the use of responseSchema for Gemini calls by upgrading the model to gemini-3-pro-preview, aiming to achieve structured JSON output. However, it introduces potential prompt injection vulnerabilities by directly embedding untrusted user input (videoUrl, videoTitle) into the LLM prompts and system instructions. These inputs must be validated and sanitized to prevent attackers from manipulating the LLM's behavior, especially given its access to the googleSearch tool. Additionally, I've provided suggestions to improve robustness and consistency, such as handling potential empty string responses before JSON.parse, ensuring consistent use of as const for schema definitions, and adding the temperature parameter for predictable output.

[gemini-code-assist]

JSON.parse('') will throw an error if the API returns an empty string. To make this more robust, it's safer to parse a fallback empty object, a pattern already used in gemini-video-analyzer.ts.

Suggested change

	const text = response.text ?? '';
	return JSON.parse(text);
	const text = response.text || '{}';
	return JSON.parse(text);

[gemini-code-assist]

JSON.parse('') will throw an error if the API returns an empty string. To make this more robust, it's safer to parse a fallback empty object, a pattern already used in gemini-video-analyzer.ts.

Suggested change

	const text = response.text ?? '';
	parsed = JSON.parse(text);
	const text = response.text || '{}';
	parsed = JSON.parse(text);

[gemini-code-assist]

The videoUrl and videoTitle parameters from the request body are directly embedded into the LLM prompt without sanitization. An attacker can provide a malicious URL or title containing instructions to manipulate the LLM's behavior. Since the LLM has access to the googleSearch tool, this could be used to perform arbitrary searches or exfiltrate information through search queries.

[gemini-code-assist]

The videoUrl parameter is used to construct the systemInstruction for the Gemini model. Specifically, the videoId (which can be the full videoUrl if the regex doesn't match) is embedded in the system instruction. This allows for prompt injection, where an attacker can provide a malicious URL to manipulate the LLM's behavior.

[gemini-code-assist]

For improved type safety and consistency with gemini-video-analyzer.ts, consider adding as const to all required arrays within this schema definition. This makes the types stricter and prevents accidental modifications.

For example:

// ...
        required: ['type', 'title', 'description', 'priority'] as const,
// ...

[gemini-code-assist]

The temperature setting is omitted here, while other Gemini API calls in the codebase specify it (e.g., 0.3 in extract-events/route.ts, 0.2 in transcribe/route.ts). To ensure consistent and predictable model behavior for structured data extraction, it's recommended to explicitly set a temperature here as well. A low value like 0.3 would be consistent with other similar calls.

    config: {
      systemInstruction,
      responseMimeType: 'application/json',
      responseSchema,
      tools: [{ googleSearch: {} }],
      temperature: 0.3,
    },

[Copilot]

Pull request overview

Restores the PK=998 “controlled generation + search grounding” pattern for Gemini calls by reintroducing responseMimeType: application/json + responseSchema alongside googleSearch, using gemini-3-pro-preview to avoid the Vertex AI limitation seen in PR #48.

Changes:

• Reintroduced a full responseSchema (Type-system) + responseMimeType in the Gemini video analyzer, including the new e22Snippets output field.
• Updated /api/extract-events Gemini paths to use responseSchema + responseMimeType + googleSearch and upgraded the model.
• Upgraded /api/transcribe Gemini model to gemini-3-pro-preview.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.

File	Description
apps/web/src/lib/gemini-video-analyzer.ts	Restores PK=998 structured output (schema + JSON mime) and adds e22Snippets; switches model to gemini-3-pro-preview.
apps/web/src/app/api/extract-events/route.ts	Restores Gemini structured extraction via Type response schema + JSON mime; switches Gemini model to gemini-3-pro-preview.
apps/web/src/app/api/transcribe/route.ts	Switches Gemini transcription/search grounding model to gemini-3-pro-preview.

[Copilot]

Switching transcription/search grounding from gemini-2.5-flash to gemini-3-pro-preview may significantly increase latency and cost for this endpoint, even though it doesn't use responseSchema controlled generation. If the model change is only needed for the responseSchema+googleSearch combination, consider keeping Flash here (or making the model configurable via an env var) to avoid an avoidable operational regression.

[Copilot]

extractWithGemini() is called when you already have transcript text, but it still enables googleSearch and uses the more expensive gemini-3-pro-preview to support schema+tool together. If search grounding isn't needed for transcript-only extraction (the prompt doesn't instruct it), consider removing the googleSearch tool in this path so you can use a cheaper model (or keep Pro only for the direct videoUrl search path).

Suggested change

	model: 'gemini-3-pro-preview',
	contents: `${SYSTEM_PROMPT}\n\n${buildUserPrompt(trimmed, videoTitle, videoUrl)}`,
	config: {
	temperature: 0.3,
	responseMimeType: 'application/json',
	responseSchema: geminiResponseSchema,
	tools: [{ googleSearch: {} }],
	model: 'gemini-1.5-flash-latest',
	contents: `${SYSTEM_PROMPT}\n\n${buildUserPrompt(trimmed, videoTitle, videoUrl)}`,
	config: {
	temperature: 0.3,
	responseMimeType: 'application/json',
	responseSchema: geminiResponseSchema,

[Copilot]

response.text || '{}' silently turns an empty/blocked model response into an object missing required fields, but the function still casts it to VideoAnalysisResult. This can propagate undefined values to callers and make failures hard to diagnose. Prefer throwing a descriptive error when response.text is empty/non-JSON (or validate required keys before returning).

Suggested change

	const resultText = response.text || '{}';
	return JSON.parse(resultText) as VideoAnalysisResult;
	const rawText = response.text;
	if (!rawText || !rawText.trim()) {
	throw new Error('Gemini video analysis returned empty response text');
	}
	let parsed: unknown;
	try {
	parsed = JSON.parse(rawText);
	} catch {
	throw new Error('Gemini video analysis returned non-JSON response text');
	}
	const result = parsed as Partial<VideoAnalysisResult> | null;
	if (
	!result ||
	typeof result.title !== 'string' ||
	typeof result.summary !== 'string' ||
	!Array.isArray(result.transcript) ||
	!Array.isArray(result.events) ||
	!Array.isArray(result.actions) ||
	!Array.isArray(result.topics) ||
	typeof result.architectureCode !== 'string' ||
	typeof result.ingestScript !== 'string' ||
	!Array.isArray(result.e22Snippets)
	) {
	throw new Error(
	'Gemini video analysis response is missing required fields',
	);
	}
	return result as VideoAnalysisResult;

[Copilot]

generateContent no longer sets a temperature here (it was previously set elsewhere in the codebase for similar calls). With tool use + long structured outputs, the default temperature can increase variance and failure rate. Consider explicitly setting temperature (and any other stability-related config like topP) to keep outputs deterministic and reduce retries/cost.

Suggested change

	tools: [{ googleSearch: {} }],
	tools: [{ googleSearch: {} }],
	temperature: 0.2,
	topP: 0.8,

[Copilot]

The Gemini schema allows timestamp to be omitted (required excludes it) but the OpenAI path always includes timestamp (nullable) and the prompt examples expect it. This leads to inconsistent response shapes depending on provider/model. Consider either adding timestamp to the Gemini schema required list (keeping it nullable) or updating the prompt/consumers to treat it as truly optional everywhere.

Suggested change

	required: ['type', 'title', 'description', 'priority'],
	required: ['type', 'title', 'description', 'timestamp', 'priority'],