diff --git a/.changeset/gemini-native-image-model-options.md b/.changeset/gemini-native-image-model-options.md new file mode 100644 index 000000000..15d81bf0c --- /dev/null +++ b/.changeset/gemini-native-image-model-options.md @@ -0,0 +1,15 @@ +--- +'@tanstack/ai-gemini': minor +--- + +Type Gemini-native image models with their own provider options. `GeminiImageModelProviderOptionsByName` mapped **every** image model to the Imagen-shaped `GeminiImageProviderOptions`, so `modelOptions: { safetySettings, thinkingConfig, imageConfig, systemInstruction }` was a compile error on `gemini-3.1-flash-image-preview`, `gemini-3.1-flash-lite-image`, `gemini-3-pro-image-preview`, and `gemini-2.5-flash-image` — even though those models are served by `generateContent`, whose `GenerateContentConfig` accepts all of them. The adapter compensated by forwarding only `seed`, silently dropping anything else. + +The map now splits native vs Imagen, mirroring the split already used by `GeminiImageModelSizeByName` and `GeminiImageModelInputModalitiesByName`: native models get the new `GeminiNativeImageProviderOptions` (`seed`, `safetySettings`, `thinkingConfig`, `imageConfig`, `systemInstruction`), Imagen models keep `GeminiImageProviderOptions`. Both API paths now pick their config fields by name — never a wholesale spread — so neither shape's fields can reach the other's endpoint. Runtime routing moves the same way, off a `gemini-` prefix test onto membership in `GEMINI_NATIVE_IMAGE_MODELS`: a `gemini-*` image model not present in that list now routes to `generateImages` instead of `generateContent`, so it fails against that endpoint rather than silently taking the native path. + +`responseModalities` stays a protected adapter default (`['TEXT', 'IMAGE']`) and is deliberately absent from the new type. `modelOptions.imageConfig` merges **over** the `imageConfig` derived from the portable `size` option, per field — passing only `imageConfig.imageSize` keeps the `aspectRatio` that `size` implied. `HarmCategory` and `HarmBlockThreshold` are now re-exported so `safetySettings` can be written without adding `@google/genai` to your own dependencies. + +`GEMINI_NATIVE_IMAGE_MODELS` and `isGeminiNativeImageModel` are exported so callers can read the same list the adapter uses for routing. + +Native `imageConfig` is now `GeminiNativeImageConfig`: only `aspectRatio` and `imageSize`. Other `@google/genai` `ImageConfig` keys type-checked and then threw on the Gemini Developer API. + +**BREAKING (types only):** Imagen fields no longer compile on Gemini-native image models — `aspectRatio`, `negativePrompt`, `personGeneration`, `safetyFilterLevel`, `addWatermark`, `language`, `outputMimeType`, `outputCompressionQuality`, `guidanceScale`, `enhancePrompt`, `includeSafetyAttributes`, `includeRaiReason`, `outputGcsUri`, `labels`. They previously type-checked but were already dropped at runtime (only `seed` was ever forwarded to `generateContent`), so no request behaviour changes. The compiler now reports what was already happening. Migrate `aspectRatio` to the portable `size` option (`'16:9_4K'`) or to `modelOptions.imageConfig`, and drop the rest. Native `imageConfig` also no longer accepts Vertex-only SDK keys such as `personGeneration` and `outputMimeType`. `GeminiImageAdapter.generateImages` (and its `~types.providerOptions`) also widens from `ImageGenerationOptions` to `ImageGenerationOptions`, which affects code structurally annotated against the old signature. diff --git a/docs/adapters/gemini.md b/docs/adapters/gemini.md index 54298b152..d41a61900 100644 --- a/docs/adapters/gemini.md +++ b/docs/adapters/gemini.md @@ -414,7 +414,10 @@ The Gemini adapter supports two types of image generation: - **Gemini native image models** (NanoBanana) — Use the `generateContent` API with models like `gemini-3.1-flash-image`. These support aspect ratio control plus resolution tiers (`512`, `1K`, `2K`, `4K`); which ratios and tiers are accepted varies per model and is enforced at compile time. - **Imagen models** — Use the `generateImages` API with models like `imagen-4.0-generate-001`. These are dedicated image generation models with WIDTHxHEIGHT sizing. -The adapter automatically routes to the correct API based on the model name — models starting with `gemini-` use `generateContent`, while `imagen-` models use `generateImages`. +The adapter routes to `generateContent` when the model is in +`GEMINI_NATIVE_IMAGE_MODELS`. Imagen models, and any id this package does not +know, use `generateImages`. Import the list or `isGeminiNativeImageModel` +from `@tanstack/ai-gemini`. ### Example: Gemini Native Image Generation (NanoBanana) @@ -502,6 +505,10 @@ const result = await generateImage({ ### Image Model Options +`modelOptions` is typed per model family, because the two families hit different APIs. + +Imagen models (`generateImages`) take `GenerateImagesConfig` fields: + ```typescript ignore import { generateImage } from "@tanstack/ai"; import { geminiImage } from "@tanstack/ai-gemini"; @@ -517,6 +524,29 @@ const result = await generateImage({ }); ``` +Gemini native models (`generateContent`) take `seed`, `safetySettings`, +`thinkingConfig`, `imageConfig`, and `systemInstruction`. +`imageConfig` accepts only `aspectRatio` and `imageSize` on the Gemini +Developer API. + +```typescript +import { generateImage } from "@tanstack/ai"; +import { geminiImage } from "@tanstack/ai-gemini"; + +const result = await generateImage({ + adapter: geminiImage("gemini-3.1-flash-image"), + prompt: "...", + size: "16:9_4K", + modelOptions: { + thinkingConfig: { thinkingBudget: 512 }, + // Merged over the imageConfig derived from `size`, per field. + imageConfig: { imageSize: "2K" }, + }, +}); +``` + +See [Image Generation](../media/image-generation) for the full native option list. + ## Text-to-Speech (Experimental) > **Note:** Gemini TTS is experimental and may require the Live API for full functionality. @@ -604,7 +634,7 @@ Creates a Gemini summarization adapter. ### `geminiImage(model, config?)` / `createGeminiImage(model, apiKey, config?)` -Creates a Gemini image adapter. Automatically routes to the correct API based on the model name — `gemini-*` models use `generateContent`, `imagen-*` models use `generateImages`. +Creates a Gemini image adapter. Models in `GEMINI_NATIVE_IMAGE_MODELS` use `generateContent`. Imagen models, and any unknown id, use `generateImages`. ### `geminiSpeech(model, config?)` / `createGeminiSpeech(model, apiKey, config?)` diff --git a/docs/config.json b/docs/config.json index b6f335946..32f9ffab3 100644 --- a/docs/config.json +++ b/docs/config.json @@ -446,7 +446,7 @@ "label": "Image Generation", "to": "media/image-generation", "addedAt": "2026-04-15", - "updatedAt": "2026-08-14" + "updatedAt": "2026-08-18" }, { "label": "Video Generation", @@ -811,7 +811,7 @@ "label": "Google Gemini", "to": "adapters/gemini", "addedAt": "2026-04-15", - "updatedAt": "2026-08-14" + "updatedAt": "2026-08-18" }, { "label": "Ollama", diff --git a/docs/media/image-generation.md b/docs/media/image-generation.md index 6fed5f85b..c4918e6c8 100644 --- a/docs/media/image-generation.md +++ b/docs/media/image-generation.md @@ -642,7 +642,7 @@ const result = await generateImage({ #### Gemini Native Model Options (NanoBanana) -Gemini native image models accept `GenerateContentConfig` options directly in `modelOptions`: +Gemini native image models are served by `generateContent`, so their `modelOptions` are `GenerateContentConfig` fields — a different shape from the Imagen options above: ```typescript import { generateImage } from "@tanstack/ai"; @@ -652,9 +652,45 @@ const result = await generateImage({ adapter: geminiImage("gemini-3.1-flash-image"), prompt: "A beautiful garden", size: "16:9_4K", + modelOptions: { + seed: 42, + thinkingConfig: { thinkingBudget: 512 }, + systemInstruction: "Always render in watercolor.", + // Merged over the imageConfig derived from `size`, per field. This keeps + // the 16:9 aspect ratio and overrides only the resolution tier. + // imageConfig accepts only aspectRatio and imageSize on the Gemini + // Developer API. + imageConfig: { imageSize: "2K" }, + }, }); ``` +`safetySettings` takes the SDK's `HarmCategory` / `HarmBlockThreshold` enums, so plain strings won't type-check. Both are re-exported from `@tanstack/ai-gemini` — you don't need `@google/genai` in your own dependencies: + +```typescript +import { generateImage } from "@tanstack/ai"; +import { + HarmBlockThreshold, + HarmCategory, + geminiImage, +} from "@tanstack/ai-gemini"; + +const result = await generateImage({ + adapter: geminiImage("gemini-3.1-flash-image-preview"), + prompt: "A beautiful garden", + modelOptions: { + safetySettings: [ + { + category: HarmCategory.HARM_CATEGORY_HATE_SPEECH, + threshold: HarmBlockThreshold.BLOCK_ONLY_HIGH, + }, + ], + }, +}); +``` + +`responseModalities` is not accepted — the adapter always requests `['TEXT', 'IMAGE']`, so nothing can silently disable image output. + ### Response Format The image generation result includes: diff --git a/packages/ai-gemini/src/adapters/image.ts b/packages/ai-gemini/src/adapters/image.ts index 521e16e47..4c5629a3e 100644 --- a/packages/ai-gemini/src/adapters/image.ts +++ b/packages/ai-gemini/src/adapters/image.ts @@ -7,6 +7,7 @@ import { } from '../utils' import { buildGeminiUsage } from '../usage' import { + isGeminiNativeImageModel, parseNativeImageSize, sizeToAspectRatio, validateImageSize, @@ -15,10 +16,11 @@ import { } from '../image/image-provider-options' import type { GeminiImageModels } from '../model-meta' import type { + GeminiAnyImageProviderOptions, GeminiImageModelInputModalitiesByName, GeminiImageModelProviderOptionsByName, GeminiImageModelSizeByName, - GeminiImageProviderOptions, + GeminiNativeImageProviderOptions, } from '../image/image-provider-options' import type { GeneratedImage, @@ -35,6 +37,7 @@ import type { GenerateImagesConfig, GenerateImagesResponse, GoogleGenAI, + ImageConfig, Part, } from '@google/genai' import type { GeminiClientConfig } from '../utils/client' @@ -65,7 +68,7 @@ export class GeminiImageAdapter< TModel extends GeminiImageModel, > extends BaseImageAdapter< TModel, - GeminiImageProviderOptions, + GeminiAnyImageProviderOptions, GeminiImageModelProviderOptionsByName, GeminiImageModelSizeByName, GeminiImageModelInputModalitiesByName @@ -75,7 +78,7 @@ export class GeminiImageAdapter< // Type-only property - never assigned at runtime declare '~types': { - providerOptions: GeminiImageProviderOptions + providerOptions: GeminiAnyImageProviderOptions modelProviderOptionsByName: GeminiImageModelProviderOptionsByName modelSizeByName: GeminiImageModelSizeByName modelInputModalitiesByName: GeminiImageModelInputModalitiesByName @@ -89,7 +92,7 @@ export class GeminiImageAdapter< } async generateImages( - options: ImageGenerationOptions, + options: ImageGenerationOptions, ): Promise { const { model, logger } = options @@ -121,7 +124,7 @@ export class GeminiImageAdapter< ) } - if (this.isGeminiImageModel(model)) { + if (isGeminiNativeImageModel(model)) { return await this.generateWithGeminiApi(options, resolved) } @@ -155,29 +158,40 @@ export class GeminiImageAdapter< } } - private isGeminiImageModel(model: string): boolean { - return model.startsWith('gemini-') - } - private async generateWithGeminiApi( - options: ImageGenerationOptions, + options: ImageGenerationOptions, resolved: ResolvedMediaPrompt, ): Promise { const { model, size, numberOfImages, modelOptions } = options const parsedSize = size ? parseNativeImageSize(size) : undefined - // GeminiImageProviderOptions is Imagen-shaped — most fields - // (personGeneration, safetyFilterLevel, addWatermark, outputMimeType, - // outputCompressionQuality, guidanceScale, enhancePrompt, - // includeSafetyAttributes, includeRaiReason, outputGcsUri, labels, - // negativePrompt, language) are only valid on GenerateImagesConfig and - // would be rejected by the Gemini-native generateContent path. Pick only - // the fields that are valid on GenerateContentConfig instead of spreading - // the whole options object. - const nativeConfig: GenerateContentConfig = {} - if (modelOptions?.seed !== undefined) { - nativeConfig.seed = modelOptions.seed + // The portable `size` option is the baseline; modelOptions.imageConfig is + // the provider escape hatch and wins per field, so a caller passing only + // `imageConfig.imageSize` keeps the aspectRatio derived from `size`. + const imageConfig: ImageConfig = { + ...(parsedSize?.aspectRatio && { aspectRatio: parsedSize.aspectRatio }), + ...(parsedSize?.resolution && { imageSize: parsedSize.resolution }), + ...modelOptions?.imageConfig, + } + + // Named picks, never a wholesale spread: the Imagen-shaped fields of + // GeminiImageProviderOptions (personGeneration, safetyFilterLevel, + // addWatermark, outputMimeType, …) are only valid on GenerateImagesConfig + // and would be rejected by generateContent. Picking by name means no + // Imagen field can reach this path even if one slips past the per-model + // provider-options map. + const nativeConfig: GenerateContentConfig = { + ...(modelOptions?.seed !== undefined && { seed: modelOptions.seed }), + ...(modelOptions?.safetySettings !== undefined && { + safetySettings: modelOptions.safetySettings, + }), + ...(modelOptions?.thinkingConfig !== undefined && { + thinkingConfig: modelOptions.thinkingConfig, + }), + ...(modelOptions?.systemInstruction !== undefined && { + systemInstruction: modelOptions.systemInstruction, + }), } const config: GenerateContentConfig = { @@ -186,16 +200,7 @@ export class GeminiImageAdapter< // IMPORTANT: responseModalities is a protected default — set it AFTER // nativeConfig so nothing can silently disable image output. responseModalities: ['TEXT', 'IMAGE'], - ...(parsedSize && { - imageConfig: { - ...(parsedSize.aspectRatio && { - aspectRatio: parsedSize.aspectRatio, - }), - ...(parsedSize.resolution && { - imageSize: parsedSize.resolution, - }), - }, - }), + ...(Object.keys(imageConfig).length > 0 && { imageConfig }), } const contents = this.buildContents(resolved, numberOfImages) @@ -321,7 +326,7 @@ export class GeminiImageAdapter< } private buildImagenConfig( - options: ImageGenerationOptions, + options: ImageGenerationOptions, ): GenerateImagesConfig { const { size, numberOfImages, modelOptions } = options @@ -329,11 +334,62 @@ export class GeminiImageAdapter< // vendor `GenerateImagesConfig` fields are `field?: T` (no `| undefined`), // so we can only assign the property when we actually have a value. const sizeAspectRatio = size ? sizeToAspectRatio(size) : undefined + + // Named picks, never a wholesale spread — the mirror image of the native + // path below. A native-only field (safetySettings, thinkingConfig, + // imageConfig, systemInstruction) belongs to GenerateContentConfig and is + // rejected by generateImages with 400 INVALID_ARGUMENT, so it must not be + // able to reach here even when the caller's `modelOptions` was typed + // against both shapes at once (e.g. an adapter inferred from a union of + // model names). return { numberOfImages: numberOfImages ?? 1, - // Map size to aspect ratio if provided (modelOptions.aspectRatio will override) + // Map size to aspect ratio if provided; modelOptions.aspectRatio, + // picked after it, overrides. ...(sizeAspectRatio !== undefined && { aspectRatio: sizeAspectRatio }), - ...modelOptions, + ...(modelOptions?.aspectRatio !== undefined && { + aspectRatio: modelOptions.aspectRatio, + }), + ...(modelOptions?.personGeneration !== undefined && { + personGeneration: modelOptions.personGeneration, + }), + ...(modelOptions?.safetyFilterLevel !== undefined && { + safetyFilterLevel: modelOptions.safetyFilterLevel, + }), + ...(modelOptions?.seed !== undefined && { seed: modelOptions.seed }), + ...(modelOptions?.addWatermark !== undefined && { + addWatermark: modelOptions.addWatermark, + }), + ...(modelOptions?.language !== undefined && { + language: modelOptions.language, + }), + ...(modelOptions?.negativePrompt !== undefined && { + negativePrompt: modelOptions.negativePrompt, + }), + ...(modelOptions?.outputMimeType !== undefined && { + outputMimeType: modelOptions.outputMimeType, + }), + ...(modelOptions?.outputCompressionQuality !== undefined && { + outputCompressionQuality: modelOptions.outputCompressionQuality, + }), + ...(modelOptions?.guidanceScale !== undefined && { + guidanceScale: modelOptions.guidanceScale, + }), + ...(modelOptions?.enhancePrompt !== undefined && { + enhancePrompt: modelOptions.enhancePrompt, + }), + ...(modelOptions?.includeSafetyAttributes !== undefined && { + includeSafetyAttributes: modelOptions.includeSafetyAttributes, + }), + ...(modelOptions?.includeRaiReason !== undefined && { + includeRaiReason: modelOptions.includeRaiReason, + }), + ...(modelOptions?.outputGcsUri !== undefined && { + outputGcsUri: modelOptions.outputGcsUri, + }), + ...(modelOptions?.labels !== undefined && { + labels: modelOptions.labels, + }), } } diff --git a/packages/ai-gemini/src/image/image-provider-options.ts b/packages/ai-gemini/src/image/image-provider-options.ts index f21fd85de..7fc386a1c 100644 --- a/packages/ai-gemini/src/image/image-provider-options.ts +++ b/packages/ai-gemini/src/image/image-provider-options.ts @@ -1,12 +1,24 @@ import type { GeminiImageModels } from '../model-meta' import type { + ContentUnion, + ImageConfig, ImagePromptLanguage, PersonGeneration, SafetyFilterLevel, + SafetySetting, + ThinkingConfig, } from '@google/genai' // Re-export SDK types so users can use them directly -export type { ImagePromptLanguage, PersonGeneration, SafetyFilterLevel } +export type { + ContentUnion, + ImageConfig, + ImagePromptLanguage, + PersonGeneration, + SafetyFilterLevel, + SafetySetting, + ThinkingConfig, +} /** * Gemini Imagen aspect ratio options @@ -121,11 +133,80 @@ export interface GeminiImageProviderOptions { } /** - * Model-specific provider options mapping - * Currently all Imagen models use the same options structure + * Provider options for Gemini native image models (Nano Banana and friends). + * + * These models are served by `generateContent`, not `generateImages`, so they + * are configured by @google/genai's `GenerateContentConfig` — a different + * shape from the Imagen-only {@link GeminiImageProviderOptions} above. Only + * the `GenerateContentConfig` fields with clear image-generation semantics are + * surfaced; sampling knobs (`temperature`, `topK`, …) and chat-only plumbing + * (`tools`, `responseSchema`, …) are deliberately left out. + * + * `responseModalities` is intentionally absent: the adapter always requests + * `['TEXT', 'IMAGE']`, and letting a caller override it would silently disable + * image output on an image-generation call. + */ +export interface GeminiNativeImageProviderOptions { + /** + * Optional seed for reproducible image generation + * When the same seed is used with the same prompt and settings, + * you should get similar (though not identical) results + */ + seed?: number + + /** + * Per-category safety thresholds applied to the request + * Each entry pairs a HarmCategory with a HarmBlockThreshold + */ + safetySettings?: Array + + /** + * Controls the model's internal reasoning before it emits an image + * Use to raise or disable the thinking budget on models that support it + */ + thinkingConfig?: ThinkingConfig + + /** + * Native image output controls. Merged over the values derived from the + * portable `size` option, so fields set here win per field while the rest + * of `size` is preserved. + * + * Only `aspectRatio` and `imageSize` are accepted on the Gemini Developer + * API. Other SDK `ImageConfig` keys throw on this surface. + */ + imageConfig?: GeminiNativeImageConfig + + /** + * System-level instructions that steer the model for the whole request, + * e.g. a house art direction applied on top of the per-call prompt + */ + systemInstruction?: ContentUnion +} + +/** + * Every provider-option field this adapter understands, across both API + * paths. Used as the adapter's base (model-agnostic) option type; the + * per-model map below is what narrows a given model to the half that + * actually applies to it. + */ +export type GeminiAnyImageProviderOptions = GeminiImageProviderOptions & + GeminiNativeImageProviderOptions + +/** + * Model-specific provider options mapping. + * Gemini native image models go through `generateContent` and take + * `GenerateContentConfig` fields; Imagen models go through `generateImages` + * and take `GenerateImagesConfig` fields. Mirrors the native/Imagen split in + * {@link GeminiImageModelSizeByName} and + * {@link GeminiImageModelInputModalitiesByName}. */ export type GeminiImageModelProviderOptionsByName = { - [K in GeminiImageModels]: GeminiImageProviderOptions + [K in GeminiNativeImageModels]: GeminiNativeImageProviderOptions +} & { + [K in Exclude< + GeminiImageModels, + GeminiNativeImageModels + >]: GeminiImageProviderOptions } /** @@ -216,6 +297,16 @@ export type Gemini3ProImageSize = */ export type Gemini25FlashImageSize = GeminiStandardImageAspectRatio +/** + * `imageConfig` fields the Gemini Developer API accepts on `generateContent`. + * Other `@google/genai` `ImageConfig` keys (`personGeneration`, + * `outputMimeType`, and more) throw on this surface. + */ +export type GeminiNativeImageConfig = { + aspectRatio?: GeminiExtendedImageAspectRatio + imageSize?: '512' | '1K' | '2K' | '4K' +} + /** * Any size accepted by any Gemini native image model. Prefer the per-model * narrowing in {@link GeminiImageModelSizeByName} — this union is the widest @@ -231,14 +322,49 @@ export type GeminiNativeImageSize = * Gemini native image models that use the generateContent API path. * These models take an aspect-ratio-based size rather than Imagen's * WIDTHxHEIGHT pixel strings. + * + * This array is the single source of truth for the native/Imagen split: the + * `GeminiNativeImageModels` union and the per-model option/size/modality maps + * all derive from it. The `satisfies` clause makes a typo (or a name that + * is not a known image model) a build error rather than a phantom key on every + * per-model map. + * + * It is also the single source of truth for the adapter's runtime routing + * — see {@link isGeminiNativeImageModel}. Adding a new `gemini-*` image model + * means adding it here as well as to `GEMINI_IMAGE_MODELS` in model-meta. + * Until it is listed here it routes to the Imagen API instead and fails + * loudly on the first call, rather than silently taking the wrong option + * shape. */ +export const GEMINI_NATIVE_IMAGE_MODELS = [ + 'gemini-3.1-flash-image', + 'gemini-3.1-flash-image-preview', + 'gemini-3.1-flash-lite-image', + 'gemini-3-pro-image', + 'gemini-3-pro-image-preview', + 'gemini-2.5-flash-image', +] as const satisfies ReadonlyArray + export type GeminiNativeImageModels = - | 'gemini-3.1-flash-image' - | 'gemini-3.1-flash-image-preview' - | 'gemini-3.1-flash-lite-image' - | 'gemini-3-pro-image' - | 'gemini-3-pro-image-preview' - | 'gemini-2.5-flash-image' + (typeof GEMINI_NATIVE_IMAGE_MODELS)[number] + +const NATIVE_IMAGE_MODEL_NAMES: ReadonlySet = new Set( + GEMINI_NATIVE_IMAGE_MODELS, +) + +/** + * Runtime counterpart to {@link GeminiNativeImageModels} — decides which of + * the two Gemini image APIs a model goes to. + * + * Membership in {@link GEMINI_NATIVE_IMAGE_MODELS}, not a `gemini-` prefix + * test, so the runtime route and the type-level split cannot drift apart. An + * id this package does not know about reaches the Imagen endpoint and fails + * there, which is the intended signal to add the model here rather than to + * have it silently take the native path with Imagen-shaped option types. + */ +export function isGeminiNativeImageModel(model: string): boolean { + return NATIVE_IMAGE_MODEL_NAMES.has(model) +} /** * Model-specific size options mapping. Each native model gets its own ratio × diff --git a/packages/ai-gemini/src/index.ts b/packages/ai-gemini/src/index.ts index f7ed26cea..7e1e426b1 100644 --- a/packages/ai-gemini/src/index.ts +++ b/packages/ai-gemini/src/index.ts @@ -28,6 +28,9 @@ export { } from './adapters/image' export type { GeminiImageProviderOptions, + GeminiNativeImageConfig, + GeminiNativeImageProviderOptions, + GeminiAnyImageProviderOptions, GeminiImageModelProviderOptionsByName, GeminiAspectRatio, // Per-model size narrowing. `GeminiImageModelSizeByName` is the map @@ -46,7 +49,16 @@ export type { PersonGeneration, SafetyFilterLevel, ImagePromptLanguage, + SafetySetting, + ThinkingConfig, + ImageConfig, + ContentUnion, } from './image/image-provider-options' +// `SafetySetting` is built from two SDK enums, and enums are values — they +// cannot travel through `export type`. Re-exported here so `safetySettings` +// is usable with only `@tanstack/ai-gemini` installed, without the consumer +// having to add `@google/genai` to their own dependencies. +export { HarmBlockThreshold, HarmCategory } from '@google/genai' // Embedding adapter - for embedding vectors export { @@ -116,6 +128,10 @@ export { } from './model-meta' export { GEMINI_MODELS as GeminiTextModels } from './model-meta' export { GEMINI_IMAGE_MODELS as GeminiImageModels } from './model-meta' +export { + GEMINI_NATIVE_IMAGE_MODELS, + isGeminiNativeImageModel, +} from './image/image-provider-options' export { GEMINI_TTS_MODELS as GeminiTTSModels } from './model-meta' export { GEMINI_TTS_VOICES as GeminiTTSVoices } from './model-meta' export { GEMINI_AUDIO_MODELS as GeminiAudioModels } from './model-meta' diff --git a/packages/ai-gemini/tests/image-adapter.test.ts b/packages/ai-gemini/tests/image-adapter.test.ts index d503c61d7..2fecc4a7d 100644 --- a/packages/ai-gemini/tests/image-adapter.test.ts +++ b/packages/ai-gemini/tests/image-adapter.test.ts @@ -1,7 +1,15 @@ import { describe, it, expect, vi } from 'vitest' +import { + HarmBlockThreshold, + HarmCategory, + ImagePromptLanguage, + PersonGeneration, + SafetyFilterLevel, +} from '@google/genai' import { generateImage } from '@tanstack/ai' import { resolveDebugOption } from '@tanstack/ai/adapter-internals' import { GeminiImageAdapter, createGeminiImage } from '../src/adapters/image' +import { GEMINI_NATIVE_IMAGE_MODELS, isGeminiNativeImageModel } from '../src' import { parseNativeImageSize, sizeToAspectRatio, @@ -10,6 +18,55 @@ import { validatePrompt, } from '../src/image/image-provider-options' +const mockImageResponse = { + candidates: [ + { + content: { + parts: [{ inlineData: { mimeType: 'image/png', data: 'out' } }], + }, + }, + ], +} + +/** + * A native-path adapter whose `client.models.generateContent` is stubbed, so + * tests can assert the exact config object handed to the SDK. + */ +function mockedNativeAdapter() { + const mockGenerateContent = vi.fn().mockResolvedValueOnce(mockImageResponse) + const adapter = createGeminiImage( + 'gemini-3.1-flash-image-preview', + 'test-api-key', + ) + ;( + adapter as unknown as { + client: { models: { generateContent: unknown } } + } + ).client = { + models: { generateContent: mockGenerateContent }, + } + return { adapter, mockGenerateContent } +} + +/** + * An Imagen-path adapter whose `client.models.generateImages` is stubbed, so + * tests can assert the exact GenerateImagesConfig handed to the SDK. + */ +function mockedImagenAdapter() { + const mockGenerateImages = vi.fn().mockResolvedValueOnce({ + generatedImages: [{ image: { imageBytes: 'imagen-b64' } }], + }) + const adapter = createGeminiImage('imagen-4.0-generate-001', 'test-api-key') + ;( + adapter as unknown as { + client: { models: { generateImages: unknown } } + } + ).client = { + models: { generateImages: mockGenerateImages }, + } + return { adapter, mockGenerateImages } +} + describe('Gemini Image Adapter', () => { describe('createGeminiImage', () => { it('creates an adapter with the provided API key', () => { @@ -912,35 +969,199 @@ describe('Gemini Image Adapter', () => { }) }) - describe('multimodal prompt (image-conditioned generation)', () => { - const testLogger = resolveDebugOption(false) - const mockImageResponse = { - candidates: [ - { - content: { - parts: [{ inlineData: { mimeType: 'image/png', data: 'out' } }], - }, + describe('native modelOptions (GenerateContentConfig)', () => { + // Regression: GeminiImageModelProviderOptionsByName used to map every + // image model — native ones included — to the Imagen-shaped + // GeminiImageProviderOptions, so these fields were a compile error and the + // adapter whitelisted only `seed`. + const safetySettings = [ + { + category: HarmCategory.HARM_CATEGORY_HATE_SPEECH, + threshold: HarmBlockThreshold.BLOCK_ONLY_HIGH, + }, + ] + + it('forwards safetySettings to generateContent', async () => { + const { adapter, mockGenerateContent } = mockedNativeAdapter() + + await generateImage({ + adapter, + prompt: 'A quiet harbour', + modelOptions: { safetySettings }, + }) + + const args = mockGenerateContent.mock.calls[0]![0] + expect(args.config.safetySettings).toEqual(safetySettings) + }) + + it('forwards thinkingConfig to generateContent', async () => { + const { adapter, mockGenerateContent } = mockedNativeAdapter() + + await generateImage({ + adapter, + prompt: 'A quiet harbour', + modelOptions: { thinkingConfig: { thinkingBudget: 512 } }, + }) + + const args = mockGenerateContent.mock.calls[0]![0] + expect(args.config.thinkingConfig).toEqual({ thinkingBudget: 512 }) + }) + + it('forwards systemInstruction to generateContent', async () => { + const { adapter, mockGenerateContent } = mockedNativeAdapter() + + await generateImage({ + adapter, + prompt: 'A quiet harbour', + modelOptions: { systemInstruction: 'Always render in watercolor.' }, + }) + + const args = mockGenerateContent.mock.calls[0]![0] + expect(args.config.systemInstruction).toBe('Always render in watercolor.') + }) + + it('merges modelOptions.imageConfig over the size-derived imageConfig', async () => { + // `size` is the portable API, `imageConfig` the provider escape hatch: + // the overriding field wins, the untouched one survives. + const { adapter, mockGenerateContent } = mockedNativeAdapter() + + await generateImage({ + adapter, + prompt: 'A quiet harbour', + size: '16:9_4K', + modelOptions: { imageConfig: { imageSize: '2K' } }, + }) + + const args = mockGenerateContent.mock.calls[0]![0] + expect(args.config.imageConfig).toEqual({ + aspectRatio: '16:9', + imageSize: '2K', + }) + }) + + it('applies modelOptions.imageConfig when no size is given', async () => { + const { adapter, mockGenerateContent } = mockedNativeAdapter() + + await generateImage({ + adapter, + prompt: 'A quiet harbour', + modelOptions: { imageConfig: { aspectRatio: '21:9' } }, + }) + + const args = mockGenerateContent.mock.calls[0]![0] + expect(args.config.imageConfig).toEqual({ aspectRatio: '21:9' }) + }) + + it('keeps Imagen models on GenerateImagesConfig with no native fields', async () => { + const { adapter, mockGenerateImages } = mockedImagenAdapter() + + await generateImage({ + adapter, + prompt: 'A quiet harbour', + size: '1920x1080', + modelOptions: { + personGeneration: PersonGeneration.ALLOW_ADULT, + negativePrompt: 'blurry', + // Native-only fields. The per-model type rejects them (hence the + // cast, as in the responseModalities regression test above), but an + // adapter inferred from a union of model names widens modelOptions + // to both shapes, so they can still arrive here at runtime — and + // generateImages answers a GenerateContentConfig field with + // 400 INVALID_ARGUMENT. The named picks must drop them. + safetySettings: [ + { + category: HarmCategory.HARM_CATEGORY_HATE_SPEECH, + threshold: HarmBlockThreshold.BLOCK_ONLY_HIGH, + }, + ], + thinkingConfig: { thinkingBudget: 512 }, + imageConfig: { imageSize: '2K' }, + systemInstruction: 'Always render in watercolor.', + } as unknown as never, + }) + + // Exact match: the Imagen path gets its own GenerateImagesConfig fields + // and nothing else. + expect(mockGenerateImages).toHaveBeenCalledWith({ + model: 'imagen-4.0-generate-001', + prompt: 'A quiet harbour', + config: { + numberOfImages: 1, + aspectRatio: '16:9', + personGeneration: PersonGeneration.ALLOW_ADULT, + negativePrompt: 'blurry', }, - ], - } + }) + }) - function mockedNativeAdapter() { - const mockGenerateContent = vi - .fn() - .mockResolvedValueOnce(mockImageResponse) - const adapter = createGeminiImage( - 'gemini-3.1-flash-image-preview', - 'test-api-key', - ) - ;( - adapter as unknown as { - client: { models: { generateContent: unknown } } - } - ).client = { - models: { generateContent: mockGenerateContent }, - } - return { adapter, mockGenerateContent } - } + it('forwards the whole Imagen option set to generateImages', async () => { + // The Imagen path picks fields by name, so this guards against a field + // being forgotten in that list. + const { adapter, mockGenerateImages } = mockedImagenAdapter() + + await generateImage({ + adapter, + prompt: 'A quiet harbour', + modelOptions: { + aspectRatio: '21:9', + personGeneration: PersonGeneration.ALLOW_ADULT, + safetyFilterLevel: SafetyFilterLevel.BLOCK_ONLY_HIGH, + seed: 42, + addWatermark: false, + language: ImagePromptLanguage.en, + negativePrompt: 'blurry', + outputMimeType: 'image/jpeg', + outputCompressionQuality: 80, + guidanceScale: 12, + enhancePrompt: true, + includeSafetyAttributes: true, + includeRaiReason: true, + outputGcsUri: 'gs://bucket/out', + labels: { team: 'design' }, + }, + }) + + expect(mockGenerateImages).toHaveBeenCalledWith({ + model: 'imagen-4.0-generate-001', + prompt: 'A quiet harbour', + config: { + numberOfImages: 1, + aspectRatio: '21:9', + personGeneration: PersonGeneration.ALLOW_ADULT, + safetyFilterLevel: SafetyFilterLevel.BLOCK_ONLY_HIGH, + seed: 42, + addWatermark: false, + language: ImagePromptLanguage.en, + negativePrompt: 'blurry', + outputMimeType: 'image/jpeg', + outputCompressionQuality: 80, + guidanceScale: 12, + enhancePrompt: true, + includeSafetyAttributes: true, + includeRaiReason: true, + outputGcsUri: 'gs://bucket/out', + labels: { team: 'design' }, + }, + }) + }) + + it('lets modelOptions.aspectRatio override the size-derived one', async () => { + const { adapter, mockGenerateImages } = mockedImagenAdapter() + + await generateImage({ + adapter, + prompt: 'A quiet harbour', + size: '1920x1080', + modelOptions: { aspectRatio: '9:16' }, + }) + + const args = mockGenerateImages.mock.calls[0]![0] + expect(args.config.aspectRatio).toBe('9:16') + }) + }) + + describe('multimodal prompt (image-conditioned generation)', () => { + const testLogger = resolveDebugOption(false) it('maps interleaved prompt parts onto multimodal contents in order', async () => { const { adapter, mockGenerateContent } = mockedNativeAdapter() @@ -1109,3 +1330,14 @@ describe('Gemini Image Adapter', () => { }) }) }) + +describe('GEMINI_NATIVE_IMAGE_MODELS public routing list', () => { + it('exports the same membership the adapter uses', () => { + expect(isGeminiNativeImageModel('gemini-3.1-flash-image')).toBe(true) + expect(isGeminiNativeImageModel('gemini-3-pro-image')).toBe(true) + expect(isGeminiNativeImageModel('gemini-2.5-flash-image')).toBe(true) + expect(isGeminiNativeImageModel('imagen-4.0-generate-001')).toBe(false) + expect(isGeminiNativeImageModel('gemini-9-pro-image')).toBe(false) + expect(GEMINI_NATIVE_IMAGE_MODELS).toContain('gemini-3.1-flash-image') + }) +}) diff --git a/packages/ai-gemini/tests/image-per-model-type-safety.test.ts b/packages/ai-gemini/tests/image-per-model-type-safety.test.ts index 96d47a9e8..537d0369b 100644 --- a/packages/ai-gemini/tests/image-per-model-type-safety.test.ts +++ b/packages/ai-gemini/tests/image-per-model-type-safety.test.ts @@ -11,10 +11,12 @@ * `createImageOptions` is the identity helper for `generateImage()` options, * so nothing here touches the network. */ -import { describe, expectTypeOf, it } from 'vitest' +import { beforeAll, describe, expectTypeOf, it } from 'vitest' import { createImageOptions } from '@tanstack/ai' import { createGeminiImage } from '../src/adapters/image' import type { GeminiImageModelSizeByName } from '../src/image/image-provider-options' +import { geminiImage } from '../src' +import type { GeminiImageModelProviderOptionsByName } from '../src' const apiKey = 'test-api-key' @@ -238,3 +240,169 @@ describe('Gemini image size map shape assertions', () => { >() }) }) + +// Set a dummy API key so adapter construction does not throw at runtime. +// These tests only exercise compile-time type gating; no network calls are made. +beforeAll(() => { + process.env['GOOGLE_API_KEY'] = 'sk-test-dummy' +}) + +describe('Gemini per-model image modelOptions gating', () => { + describe('gemini-3.1-flash-image-preview — native (GenerateContentConfig)', () => { + it('accepts the native option set', () => { + createImageOptions({ + adapter: geminiImage('gemini-3.1-flash-image-preview'), + prompt: 'a quiet harbour', + modelOptions: { + seed: 7, + safetySettings: [], + thinkingConfig: { thinkingBudget: 512 }, + imageConfig: { aspectRatio: '16:9', imageSize: '2K' }, + systemInstruction: 'Always render in watercolor.', + }, + }) + }) + + it('rejects Vertex-only imageConfig fields', () => { + createImageOptions({ + adapter: geminiImage('gemini-3.1-flash-image-preview'), + prompt: 'a quiet harbour', + modelOptions: { + imageConfig: { + // @ts-expect-error - personGeneration throws on the Gemini Developer API + personGeneration: 'ALLOW_ADULT', + }, + }, + }) + }) + + it('rejects Imagen-only options', () => { + // The probes are plain values that are structurally valid on + // GeminiImageProviderOptions (`negativePrompt: string`, + // `aspectRatio: GeminiAspectRatio`), so the errors below come from the + // native/Imagen split and nothing else — an enum-typed field such as + // `personGeneration` would reject a string literal on either shape and + // would still "pass" with the split reverted. + createImageOptions({ + adapter: geminiImage('gemini-3.1-flash-image-preview'), + prompt: 'a quiet harbour', + modelOptions: { + // @ts-expect-error - negativePrompt is a GenerateImagesConfig (Imagen) field + negativePrompt: 'blurry', + }, + }) + + createImageOptions({ + adapter: geminiImage('gemini-3.1-flash-image-preview'), + prompt: 'a quiet harbour', + modelOptions: { + // @ts-expect-error - aspectRatio is Imagen-only; native models use size / imageConfig + aspectRatio: '16:9', + }, + }) + }) + + it('rejects responseModalities — the adapter owns it', () => { + createImageOptions({ + adapter: geminiImage('gemini-3.1-flash-image-preview'), + prompt: 'a quiet harbour', + modelOptions: { + // @ts-expect-error - responseModalities is a protected adapter default + responseModalities: ['TEXT'], + }, + }) + }) + }) + + describe('imagen-4.0-generate-001 — Imagen (GenerateImagesConfig)', () => { + it('accepts the Imagen option set', () => { + createImageOptions({ + adapter: geminiImage('imagen-4.0-generate-001'), + prompt: 'a quiet harbour', + modelOptions: { + aspectRatio: '16:9', + negativePrompt: 'blurry', + addWatermark: true, + outputMimeType: 'image/png', + }, + }) + }) + + it('rejects native-only options', () => { + createImageOptions({ + adapter: geminiImage('imagen-4.0-generate-001'), + prompt: 'a quiet harbour', + modelOptions: { + // @ts-expect-error - safetySettings is a GenerateContentConfig (native) field + safetySettings: [], + }, + }) + }) + }) +}) + +describe('Gemini image provider options shape assertions', () => { + describe('native models take GenerateContentConfig fields', () => { + type Options = + GeminiImageModelProviderOptionsByName['gemini-3.1-flash-image-preview'] + + it('has safetySettings', () => { + expectTypeOf().toHaveProperty('safetySettings') + }) + it('has thinkingConfig', () => { + expectTypeOf().toHaveProperty('thinkingConfig') + }) + it('has imageConfig', () => { + expectTypeOf().toHaveProperty('imageConfig') + }) + it('has systemInstruction', () => { + expectTypeOf().toHaveProperty('systemInstruction') + }) + it('has seed', () => { + expectTypeOf().toHaveProperty('seed') + }) + }) + + describe('Imagen models keep GenerateImagesConfig fields', () => { + type Options = + GeminiImageModelProviderOptionsByName['imagen-4.0-generate-001'] + + it('has aspectRatio', () => { + expectTypeOf().toHaveProperty('aspectRatio') + }) + it('has personGeneration', () => { + expectTypeOf().toHaveProperty('personGeneration') + }) + it('has negativePrompt', () => { + expectTypeOf().toHaveProperty('negativePrompt') + }) + }) + + describe('every native model id resolves to the native shape', () => { + it('gemini-3.1-flash-image', () => { + expectTypeOf< + GeminiImageModelProviderOptionsByName['gemini-3.1-flash-image'] + >().toHaveProperty('imageConfig') + }) + it('gemini-3-pro-image', () => { + expectTypeOf< + GeminiImageModelProviderOptionsByName['gemini-3-pro-image'] + >().toHaveProperty('imageConfig') + }) + it('gemini-3.1-flash-lite-image', () => { + expectTypeOf< + GeminiImageModelProviderOptionsByName['gemini-3.1-flash-lite-image'] + >().toHaveProperty('imageConfig') + }) + it('gemini-3-pro-image-preview', () => { + expectTypeOf< + GeminiImageModelProviderOptionsByName['gemini-3-pro-image-preview'] + >().toHaveProperty('imageConfig') + }) + it('gemini-2.5-flash-image', () => { + expectTypeOf< + GeminiImageModelProviderOptionsByName['gemini-2.5-flash-image'] + >().toHaveProperty('imageConfig') + }) + }) +}) diff --git a/testing/e2e/global-setup.ts b/testing/e2e/global-setup.ts index b199db906..bae3b021a 100644 --- a/testing/e2e/global-setup.ts +++ b/testing/e2e/global-setup.ts @@ -82,18 +82,9 @@ export default async function globalSetup() { mock.mount('/api/embed', ollamaEmbedMount()) mock.mount('/mistral', mistralEmbeddingsMount()) - // Gemini native image generation (#1104: GA model ids + per-model - // aspectRatio/imageSize). Like TTS above, this hits generateContent, but - // aimock's handleGemini has no image-response branch at all (it imports - // isTextResponse/isToolCallResponse/isContentWithToolCallsResponse/ - // isAudioResponse from helpers.js — isImageResponse is wired only into - // images.js's OpenAI /v1/images/* and Gemini's :predict handling) — a - // fixture shaped `{image}`/`{images}` falls through every branch and 500s - // with "Fixture response did not match any known type". Shares the - // '/v1beta/models' prefix with the Veo and batch-embed mounts above; only - // the two model paths the #1104 spec exercises are handled here, so - // everything else still falls through to those mounts / aimock's native - // Gemini handlers. + // Gemini native image generation. aimock has no generateContent image + // branch. One mount covers the #1104 size wire (aspectRatio / imageSize) + // and the #1103 modelOptions wire (safetySettings / thinkingConfig). mock.mount('/v1beta/models', geminiNativeImageMount()) // Gemini Omni Flash video generation (Interactions API). aimock handles @@ -547,12 +538,14 @@ function rejectGeminiImageRequest( } /** - * Mounts Gemini native `generateContent` image calls for the two models the - * spec uses: `gemini-3.1-flash-image` and `gemini-2.5-flash-image`. + * Mounts Gemini native `generateContent` image calls for + * `gemini-3.1-flash-image` and `gemini-2.5-flash-image`. * - * aimock has no `generateContent` image branch. This mount reads - * `generationConfig.imageConfig` and checks `aspectRatio` / `imageSize` - * for those two models. + * aimock has no `generateContent` image branch. This mount reads the raw + * body so two specs can assert on the wire: + * - `/api/gemini-image-ga-models` checks `imageConfig.aspectRatio` / `imageSize` + * - `/api/gemini-native-image-wire` checks top-level `safetySettings` and + * `generationConfig.thinkingConfig` on `gemini-2.5-flash-image` */ function geminiNativeImageMount(): Mountable { const NATIVE_IMAGE_MODELS = new Set([ @@ -560,12 +553,27 @@ function geminiNativeImageMount(): Mountable { 'gemini-2.5-flash-image', ]) - // Deliberately coupled to the `size` values in - // testing/e2e/src/routes/api.gemini-image-ga-models.ts ('16:9_2K' and - // '16:9') — a future edit to one must update the other. + // Coupled to the `size` values in api.gemini-image-ga-models.ts. const EXPECTED_ASPECT_RATIO = '16:9' const EXPECTED_FLASH_IMAGE_SIZE = '2K' + // Imagen-only GenerateImagesConfig fields must never appear on generateContent. + const IMAGEN_ONLY_FIELDS = [ + 'personGeneration', + 'safetyFilterLevel', + 'addWatermark', + 'language', + 'negativePrompt', + 'outputMimeType', + 'outputCompressionQuality', + 'guidanceScale', + 'enhancePrompt', + 'includeSafetyAttributes', + 'includeRaiReason', + 'outputGcsUri', + 'aspectRatio', + ] + return { async handleRequest( req: http.IncomingMessage, @@ -589,28 +597,64 @@ function geminiNativeImageMount(): Mountable { const imageConfig = asRecord(generationConfig?.imageConfig) const aspectRatio = imageConfig?.aspectRatio const imageSize = imageConfig?.imageSize - - if (aspectRatio !== EXPECTED_ASPECT_RATIO) { + const leaked = IMAGEN_ONLY_FIELDS.find( + (name) => + name in body || (generationConfig && name in generationConfig), + ) + if (leaked) { return rejectGeminiImageRequest( res, - `${model}: generationConfig.imageConfig.aspectRatio must be "${EXPECTED_ASPECT_RATIO}", got ${JSON.stringify(aspectRatio)}.`, + `Imagen-only field "${leaked}" reached generateContent — GenerateImagesConfig and GenerateContentConfig got crossed.`, ) } - if (model === 'gemini-3.1-flash-image') { - if (imageSize !== EXPECTED_FLASH_IMAGE_SIZE) { + const hasModelOptions = + Array.isArray(body.safetySettings) || + (generationConfig !== undefined && + typeof generationConfig.thinkingConfig === 'object' && + generationConfig.thinkingConfig !== null) + + if (model === 'gemini-2.5-flash-image' && hasModelOptions) { + if ( + !Array.isArray(body.safetySettings) || + body.safetySettings.length === 0 + ) { return rejectGeminiImageRequest( res, - `${model}: generationConfig.imageConfig.imageSize must be "${EXPECTED_FLASH_IMAGE_SIZE}", got ${JSON.stringify(imageSize)}.`, + 'Missing top-level safetySettings (modelOptions.safetySettings did not reach the wire).', + ) + } + if ( + !generationConfig || + typeof generationConfig.thinkingConfig !== 'object' || + generationConfig.thinkingConfig === null + ) { + return rejectGeminiImageRequest( + res, + 'Missing generationConfig.thinkingConfig (modelOptions.thinkingConfig did not reach the wire).', + ) + } + } else { + if (aspectRatio !== EXPECTED_ASPECT_RATIO) { + return rejectGeminiImageRequest( + res, + `${model}: generationConfig.imageConfig.aspectRatio must be "${EXPECTED_ASPECT_RATIO}", got ${JSON.stringify(aspectRatio)}.`, + ) + } + + if (model === 'gemini-3.1-flash-image') { + if (imageSize !== EXPECTED_FLASH_IMAGE_SIZE) { + return rejectGeminiImageRequest( + res, + `${model}: generationConfig.imageConfig.imageSize must be "${EXPECTED_FLASH_IMAGE_SIZE}", got ${JSON.stringify(imageSize)}.`, + ) + } + } else if (imageSize !== undefined) { + return rejectGeminiImageRequest( + res, + `${model}: generationConfig.imageConfig.imageSize must be absent.`, ) } - } else if (imageSize !== undefined) { - // gemini-2.5-flash-image: Google documents no image_size for this - // model (#1104) — the adapter must send a bare aspect ratio only. - return rejectGeminiImageRequest( - res, - `${model}: generationConfig.imageConfig.imageSize must be absent.`, - ) } res.statusCode = 200 diff --git a/testing/e2e/src/routeTree.gen.ts b/testing/e2e/src/routeTree.gen.ts index 0cadb398d..dc6e573c2 100644 --- a/testing/e2e/src/routeTree.gen.ts +++ b/testing/e2e/src/routeTree.gen.ts @@ -60,6 +60,7 @@ import { Route as ApiImageRouteImport } from './routes/api.image' import { Route as ApiGenerationPersistenceServerRouteImport } from './routes/api.generation-persistence-server' import { Route as ApiGenerationPersistenceResumeRouteImport } from './routes/api.generation-persistence-resume' import { Route as ApiGeminiImageGaModelsRouteImport } from './routes/api.gemini-image-ga-models' +import { Route as ApiGeminiNativeImageWireRouteImport } from './routes/api.gemini-native-image-wire' import { Route as ApiForeignInterruptRouteImport } from './routes/api.foreign-interrupt' import { Route as ApiEmbeddingRouteImport } from './routes/api.embedding' import { Route as ApiDurableTakeoverRouteImport } from './routes/api.durable-takeover' @@ -342,6 +343,12 @@ const ApiGeminiImageGaModelsRoute = ApiGeminiImageGaModelsRouteImport.update({ path: '/api/gemini-image-ga-models', getParentRoute: () => rootRouteImport, } as any) +const ApiGeminiNativeImageWireRoute = + ApiGeminiNativeImageWireRouteImport.update({ + id: '/api/gemini-native-image-wire', + path: '/api/gemini-native-image-wire', + getParentRoute: () => rootRouteImport, + } as any) const ApiForeignInterruptRoute = ApiForeignInterruptRouteImport.update({ id: '/api/foreign-interrupt', path: '/api/foreign-interrupt', @@ -460,6 +467,7 @@ export interface FileRoutesByFullPath { '/api/embedding': typeof ApiEmbeddingRoute '/api/foreign-interrupt': typeof ApiForeignInterruptRoute '/api/gemini-image-ga-models': typeof ApiGeminiImageGaModelsRoute + '/api/gemini-native-image-wire': typeof ApiGeminiNativeImageWireRoute '/api/generation-persistence-resume': typeof ApiGenerationPersistenceResumeRoute '/api/generation-persistence-server': typeof ApiGenerationPersistenceServerRoute '/api/image': typeof ApiImageRouteWithChildren @@ -530,6 +538,7 @@ export interface FileRoutesByTo { '/api/embedding': typeof ApiEmbeddingRoute '/api/foreign-interrupt': typeof ApiForeignInterruptRoute '/api/gemini-image-ga-models': typeof ApiGeminiImageGaModelsRoute + '/api/gemini-native-image-wire': typeof ApiGeminiNativeImageWireRoute '/api/generation-persistence-resume': typeof ApiGenerationPersistenceResumeRoute '/api/generation-persistence-server': typeof ApiGenerationPersistenceServerRoute '/api/image': typeof ApiImageRouteWithChildren @@ -601,6 +610,7 @@ export interface FileRoutesById { '/api/embedding': typeof ApiEmbeddingRoute '/api/foreign-interrupt': typeof ApiForeignInterruptRoute '/api/gemini-image-ga-models': typeof ApiGeminiImageGaModelsRoute + '/api/gemini-native-image-wire': typeof ApiGeminiNativeImageWireRoute '/api/generation-persistence-resume': typeof ApiGenerationPersistenceResumeRoute '/api/generation-persistence-server': typeof ApiGenerationPersistenceServerRoute '/api/image': typeof ApiImageRouteWithChildren @@ -673,6 +683,7 @@ export interface FileRouteTypes { | '/api/embedding' | '/api/foreign-interrupt' | '/api/gemini-image-ga-models' + | '/api/gemini-native-image-wire' | '/api/generation-persistence-resume' | '/api/generation-persistence-server' | '/api/image' @@ -743,6 +754,7 @@ export interface FileRouteTypes { | '/api/embedding' | '/api/foreign-interrupt' | '/api/gemini-image-ga-models' + | '/api/gemini-native-image-wire' | '/api/generation-persistence-resume' | '/api/generation-persistence-server' | '/api/image' @@ -813,6 +825,7 @@ export interface FileRouteTypes { | '/api/embedding' | '/api/foreign-interrupt' | '/api/gemini-image-ga-models' + | '/api/gemini-native-image-wire' | '/api/generation-persistence-resume' | '/api/generation-persistence-server' | '/api/image' @@ -884,6 +897,7 @@ export interface RootRouteChildren { ApiEmbeddingRoute: typeof ApiEmbeddingRoute ApiForeignInterruptRoute: typeof ApiForeignInterruptRoute ApiGeminiImageGaModelsRoute: typeof ApiGeminiImageGaModelsRoute + ApiGeminiNativeImageWireRoute: typeof ApiGeminiNativeImageWireRoute ApiGenerationPersistenceResumeRoute: typeof ApiGenerationPersistenceResumeRoute ApiGenerationPersistenceServerRoute: typeof ApiGenerationPersistenceServerRoute ApiImageRoute: typeof ApiImageRouteWithChildren @@ -1278,6 +1292,13 @@ declare module '@tanstack/react-router' { preLoaderRoute: typeof ApiGeminiImageGaModelsRouteImport parentRoute: typeof rootRouteImport } + '/api/gemini-native-image-wire': { + id: '/api/gemini-native-image-wire' + path: '/api/gemini-native-image-wire' + fullPath: '/api/gemini-native-image-wire' + preLoaderRoute: typeof ApiGeminiNativeImageWireRouteImport + parentRoute: typeof rootRouteImport + } '/api/foreign-interrupt': { id: '/api/foreign-interrupt' path: '/api/foreign-interrupt' @@ -1489,6 +1510,7 @@ const rootRouteChildren: RootRouteChildren = { ApiEmbeddingRoute: ApiEmbeddingRoute, ApiForeignInterruptRoute: ApiForeignInterruptRoute, ApiGeminiImageGaModelsRoute: ApiGeminiImageGaModelsRoute, + ApiGeminiNativeImageWireRoute: ApiGeminiNativeImageWireRoute, ApiGenerationPersistenceResumeRoute: ApiGenerationPersistenceResumeRoute, ApiGenerationPersistenceServerRoute: ApiGenerationPersistenceServerRoute, ApiImageRoute: ApiImageRouteWithChildren, diff --git a/testing/e2e/src/routes/api.gemini-native-image-wire.ts b/testing/e2e/src/routes/api.gemini-native-image-wire.ts new file mode 100644 index 000000000..94c15a246 --- /dev/null +++ b/testing/e2e/src/routes/api.gemini-native-image-wire.ts @@ -0,0 +1,71 @@ +import { createFileRoute } from '@tanstack/react-router' +import { generateImage } from '@tanstack/ai' +import { createImageAdapter } from '@/lib/media-providers' + +/** + * Wire-format verification for Gemini-native `modelOptions` on the image + * generation path (fix/gemini-native-image-model-options). + * + * Before that fix, `GeminiImageAdapter`'s `generateWithGeminiApi` only ever + * forwarded `modelOptions.seed` into the `generateContent` request — + * `safetySettings`, `thinkingConfig`, `imageConfig`, and `systemInstruction` + * were silently dropped even though the adapter's provider-options type + * (`GeminiNativeImageProviderOptions`) already declared them. This route + * drives `generateImage()` against `gemini-2.5-flash-image` with + * `modelOptions: { safetySettings, thinkingConfig }` set, hitting + * `geminiNativeImageMount` in global-setup.ts — a hand-mocked + * `POST /v1beta/models/gemini-2.5-flash-image:generateContent` endpoint that + * reads the raw, untranslated request body (aimock's own journal cannot see + * these fields for this endpoint — see that mount's comment) and rejects + * with 400 unless `safetySettings` is present at the request root and + * `generationConfig.thinkingConfig` is present nested, and rejects unless no + * Imagen-only field (`personGeneration`, `negativePrompt`, a root-level + * `aspectRatio`, …) is present anywhere in the body. + * + * A regression that stops forwarding `modelOptions` on this path — reverting + * to only `seed`, or reverting to a wholesale `...modelOptions` spread that + * lets an Imagen field cross over — makes the mount reject the request, the + * adapter's `client.models.generateContent()` call throws, and this route + * returns `ok: false`. The companion spec asserts `ok: true`. + */ +export const Route = createFileRoute('/api/gemini-native-image-wire')({ + server: { + handlers: { + POST: async () => { + const adapter = createImageAdapter('gemini') + + try { + const result = await generateImage({ + adapter, + prompt: 'a guitar in a music store', + stream: false, + modelOptions: { + safetySettings: [ + { + category: 'HARM_CATEGORY_DANGEROUS_CONTENT', + threshold: 'BLOCK_ONLY_HIGH', + }, + ], + thinkingConfig: { thinkingBudget: 128 }, + }, + }) + return new Response( + JSON.stringify({ ok: true, images: result.images.length }), + { + status: 200, + headers: { 'Content-Type': 'application/json' }, + }, + ) + } catch (error) { + return new Response( + JSON.stringify({ + ok: false, + error: error instanceof Error ? error.message : String(error), + }), + { status: 200, headers: { 'Content-Type': 'application/json' } }, + ) + } + }, + }, + }, +}) diff --git a/testing/e2e/tests/gemini-native-image-wire.spec.ts b/testing/e2e/tests/gemini-native-image-wire.spec.ts new file mode 100644 index 000000000..283638baf --- /dev/null +++ b/testing/e2e/tests/gemini-native-image-wire.spec.ts @@ -0,0 +1,43 @@ +import { test, expect } from './fixtures' + +/** + * Wire-format verification for Gemini-native `modelOptions` on the image + * generation path (fix/gemini-native-image-model-options). + * + * `/api/gemini-native-image-wire` drives `generateImage()` against + * `gemini-2.5-flash-image` with `modelOptions: { safetySettings, + * thinkingConfig }`. That request lands on `geminiNativeImageMount` in + * global-setup.ts, which reads the raw, untranslated request body (aimock's + * own journal normalises this endpoint's requests and drops these exact + * fields before journalling — see that mount's comment) and rejects with 400 + * unless `safetySettings` is present at the request root, + * `generationConfig.thinkingConfig` is present nested under + * `generationConfig`, and no Imagen-only field (`personGeneration`, + * `negativePrompt`, a root-level `aspectRatio`, …) appears anywhere in the + * body. + * + * Before the fix, `generateWithGeminiApi` only forwarded `modelOptions.seed` + * — `safetySettings` and `thinkingConfig` were silently dropped even though + * the adapter's own provider-options type already declared them. Reverting + * the fix reproduces that: the outgoing request loses both fields, the mount + * rejects it with 400, `client.models.generateContent()` throws, and the + * route returns `ok: false` — this spec's `ok` assertion fails. + */ +test.describe('gemini native image — modelOptions reach the generateContent wire', () => { + test('safetySettings and thinkingConfig survive to the request; no Imagen field does', async ({ + request, + }) => { + const res = await request.post('/api/gemini-native-image-wire') + expect(res.ok()).toBe(true) + + const { ok, images, error } = (await res.json()) as { + ok: boolean + images?: number + error?: string + } + + expect(error ?? null).toBeNull() + expect(ok).toBe(true) + expect(images).toBe(1) + }) +})