Update agent chat models and enable image/PDF multimodal support

[Al629176]

Summary

Update the agent chat configuration to use Quen 3.6 and Quen 3.6+ as the backend models, and enable multimodal support for image and PDF inputs.

Problem / Context

The chat agent needs to be routed through the new backend model configuration and support richer user inputs. Users should be able to send images and PDFs in chat and have the agent process them through the appropriate multimodal-capable model path.

Scope (optional)

Update the chat model configuration and routing so:

• Agent chat uses Quen 3.6 and Quen 3.6+ as the backend model options.
• The correct default model is selected for normal chat workloads.
• Image and PDF inputs are accepted and routed to a multimodal-capable path.
• Unsupported models/providers do not receive image/PDF payloads without a clear fallback or error.

Acceptance criteria

• Chat models updated — Agent chat configuration uses Quen 3.6 and Quen 3.6+ as backend model options.
• Default routing verified — Normal chat requests route to the intended default backend model.
• Image support enabled — Users can attach images in chat and the agent can analyze them.
• PDF support enabled — Users can attach PDFs in chat and the agent can summarize or answer questions from them.
• Capability routing safe — Image/PDF payloads only route to models/providers that support multimodal input, or show a clear actionable fallback.
• Failure messaging improved — Unsupported file type, file size, model, or provider failures do not show a generic error.
• Regression safety — E2E or integration coverage verifies normal chat, image attachment, and PDF attachment flows.
• Diff coverage >= 80% — Any implementing PR meets the changed-lines coverage gate when code changes are involved.

Related

• Agent chat model routing.
• Multimodal chat attachments for images and PDFs.

[oxoxDev]

Parity with current code (post-#1535 epic stack)

Audited the existing multimodal + provider plumbing against this issue's six ACs. Below is the exact file:line surface that's already in tree, what each AC needs, and where the gap is.

Currently shipped (covered by #1535 epic)

AC	What ships today	Where
PDF support enabled	[FILE:path/url] markers parsed, PDFs streamed through pdf_extract::extract_text_from_mem with a 60s timeout (binaries beyond extractable formats degrade to metadata-only references). Extracted text is inlined into the message content the LLM receives.	src/openhuman/agent/multimodal.rs:26 (FILE_MARKER_PREFIX), :28-32 (PDF_EXTRACTION_TIMEOUT), :982 (pdf_extract::extract_text_from_mem), :218-246 (parser loop). Shipped in PR #2954 (merged 2026-06-01).
Capability routing safe — partial	provider.supports_vision() trait method gates the agent loop: if a turn contains [IMAGE:…] markers and the active provider returns false from supports_vision(), the loop returns a ProviderCapabilityError (provider_capability Sentry tag, capability=\"vision\") instead of silently sending image markers to a text-only model. Router + reliable layers fan-out the capability check across composed providers.	src/openhuman/agent/harness/engine/core.rs:196-217 (the gate), src/openhuman/inference/provider/traits.rs:482 (default impl), :router.rs:229-232, :reliable.rs:683-686 (composite implementations).
Image attach surface (composer)	Composer accepts image/png, image/jpeg, image/webp, image/gif, image/bmp and base64-encodes into data: URIs. Marker prefix in the Rust core matches.	app/src/lib/attachments.ts:13 (ALLOWED_IMAGE_MIME_TYPES TS), src/openhuman/agent/multimodal.rs:11-19 (IMAGE_MARKER_PREFIX + ALLOWED_IMAGE_MIME_TYPES Rust).
UnsupportedFileType error variant scaffolding	PresentationError::UnsupportedFileType { extension, supported } enum variant added (Display impl asserts both fields are surfaced).	src/openhuman/tools/impl/presentation/types.rs (added on PR #3029, branch feat/2780-orchestrator-presentation-wiring, tip currently 5218a149).

Gap — what this issue closes that #1535 did not

AC	Why it's not covered today	What this issue's PR needs to ship
Image support enabled (agent can analyze them)	ChatMessage.content is a flat String (src/openhuman/inference/provider/traits.rs:13). The image marker [IMAGE:data:image/png;base64,…] is inlined as text into that string by compose_multimodal_message (multimodal.rs:309-358). Zero call sites in src/openhuman/inference/provider/ convert the marker into a provider-specific multimodal content block — grep -rn 'image_url|content_blocks' src/openhuman/inference/provider/ returns nothing. The OpenAI Chat-Completions API expects {type: \"image_url\", image_url: {url: \"data:…\"}} blocks; Anthropic expects {type: \"image\", source: {type: \"base64\", …}} blocks. The current wire format ships base64 text inside a content: String field — the model receives garbage characters, not an image. The supports_vision() gate passes the call through to the model, but the model has nothing visual to look at.	Evolve ChatMessage::content from String to a typed enum (Text(String) / Multimodal(Vec<ContentBlock>), mirroring the OpenAI/Anthropic union), wire the existing marker parser to emit content blocks instead of inline text, and teach the OpenAI-compatible + Anthropic + Responses-API request builders (src/openhuman/inference/provider/compatible_request.rs, compatible_parse.rs) to serialise the new shape. Round-trip test against a vision model fixture (small PNG → assert the request payload carries a proper image_url block, not a marker substring).
Chat models updated (Quen 3.6 / Quen 3.6+)	Separate from #1535 entirely. Model catalog + routing default is at src/openhuman/inference/types.rs (the vision_model_id/vision_state/vision_mode family) and the provider factory. No Quen 3.6 SKU is registered today.	Register qwen-3.6 and qwen-3.6-plus model entries in the provider catalog (Quen has a vision-capable variant — if Quen 3.6+ is the vision one, mark supports_vision()=true for that SKU only). Default chat routing in presets::* swaps to it.
Default routing verified	Default chat model resolution lives in src/openhuman/inference/types.rs:46-58 (presets::vision_mode_for_config).	Add a regression test pinning the default chat-model resolution against config (so a later refactor can't silently swap the default).
Failure messaging improved	PresentationError::UnsupportedFileType exists but is not constructed by execute() yet — reserved for the future format selector. For image/PDF dispatch: the only structured error today is ProviderCapabilityError (text says "this provider does not support vision input"). No actionable code for "model X doesn't support PDF" / "file too large" / "MIME rejected".	Wire UnsupportedFileType (or a sibling variant UnsupportedMime) from the actual rejection paths in multimodal.rs::parse_file_markers so the agent surfaces an actionable error to the chat surface instead of swallowing. Match shape per feedback_pr_review_terse — code + localized message + diagnostic detail.
Regression safety — image + PDF E2E	PDF path has unit coverage (multimodal.rs parser tests). Image-attachment E2E is absent because the wire format doesn't deliver to a vision model in the first place.	Vitest spec for composer → marker → core ingestion. Playwright spec for full image-attach → vision-model round-trip (mocked vision provider). PDF E2E spec to lock in #2954's behaviour.
Diff coverage ≥ 80%	Standard gate — applies to whatever this issue's implementing PR ends up doing.	Per-PR concern.

Suggested implementing-PR sequencing

1. PR A — ChatMessage::Content enum + provider request-builder updates. Pure plumbing refactor; no user-visible change yet. Lands the data shape that lets the marker resolver emit content blocks.
2. PR B — marker resolver emits ContentBlock::ImageUrl { url: data_uri } instead of [IMAGE:…] text. This is the actual delivery of images to vision models. Validated by integration test against a stub OpenAI server that asserts the request body shape.
3. PR C — Quen 3.6 / 3.6+ catalog entries + default-routing swap. Independent of A/B; can land in parallel.
4. PR D — UnsupportedMime / UnsupportedFileType wiring + Sentry redaction. Closes the failure-messaging AC.
5. PR E — E2E specs (image attach + PDF attach + vision-model routing + unsupported-MIME fallback).

PR A is the load-bearing one — without it, B/C/D ship into a wire format that drops images on the floor.

Cross-references

• Extend multimodal input to accept document and file attachments beyond images #2777 (PR feat(agent/multimodal): support [FILE:…] markers with text extraction (#2777) #2954, MERGED) — [FILE:…] markers + PDF text extraction. Covers PDF AC today.
• Surface artifact downloads and generation progress in the React UI #2779 (PR feat(chat): ArtifactCard + Tauri download + backend artifact events (#2779) #3017, MERGED) — ArtifactCard UI + artifact-download flow (the receive-side).
• Wire PPT generation and file attachments into agent definitions and error handling #2780 (PR feat(orchestrator): wire generate_presentation + require grounding (#2780) #3029, in flight) — UnsupportedFileType variant scaffolding + orchestrator grounding rule for generate_presentation.
• Files in this chat — persistent per-chat artifact registry #3024 (PR feat(chat): Files in this chat — persistent per-chat artifact panel (#3024) #3026, in flight) — Files-in-chat panel; not directly relevant to this issue.
• Artifact UX polish: native save-file dialog + ChatToolCallEvent in-progress wiring #3162 — follow-up: native save-file dialog + ChatToolCallEvent in-progress wiring + Retry-button wiring. Adjacent but distinct.
• generate_presentation: image embedding in slides via ppt-rs::add_image #3209 — follow-up: image embedding in slides via ppt-rs::add_image (output-side, not input).
• ai_list_artifacts: thread_id filter + ArtifactMeta.thread_id field #3226 — follow-up: ArtifactMeta.thread_id + ai_list_artifacts thread filter.

This issue is the input-side multimodal closer that the #1535 epic deliberately stopped short of — surface ships, wire format does not.

[sanil-23]

Image part — client pipeline implemented in #3268 (shipped gated off).

#3268 implements the full client-side image path, kept behind CHAT_ATTACHMENTS_ENABLED=false + provider vision:false because the remaining gap is backend-side:

• [IMAGE:] markers → OpenAI image_url content-array (correct multimodal wire format; text-only turns unchanged)
• image-aware token budgeting (image not counted as base64 text → no longer trimmed out of context)
• summarizer + episodic-memory hygiene (image redacted/stripped so base64 never pollutes summaries/embeddings)
• raised /rpc body limit (2 MiB → 64 MiB) so the inline image isn't rejected with 413

Verified end-to-end against a vision model (gpt-4.1-mini / gpt-5 via BYO provider) → real image description. Same image to the default chat model → empty.

Remaining (backend, blocks "Image support enabled"): the chat tier reasoning-v1 resolves server-side to DeepSeek v4 Flash (text-only). Vision is per-model, so image turns must route to a vision-capable backend model (e.g. via model_registry.vision). Once that lands, flip the flags on — no further client changes.

Not in #3268: PDF input, Quen 3.6/3.6+ model config, typed failure-messaging.

[oxoxDev]

Correction + updated remaining-work after @sanil-23's #3268 context

My earlier parity table flagged the wire-format gap as load-bearing (ChatMessage.content flat String + zero call sites converting [IMAGE:…] markers to provider multimodal content blocks). That gap is now already addressed client-side by #3268 — kept gated behind CHAT_ATTACHMENTS_ENABLED (default off, inherited from #3212 which merged 2026-06-02). Confirmed by reading the PR: it touches src/openhuman/inference/provider/compatible.rs, compatible_types.rs, compatible_tests.rs, openhuman_backend.rs. Title feat(chat): implement image attachment pipeline, gated off (#3205).

So my proposed PR-A (Content enum + request builders) and PR-B (marker → content-block resolver) collapse into landing + flag-flipping #3268. The remaining work is narrower than my first comment implied.

Refined remaining-work map

AC (from this issue)	State after #3268 lands	Still needs
Image support enabled	Wire format ready; gated false.	(a) backend model route for image turns (below), (b) flag flip.
Backend route — load-bearing blocker	reasoning-v1 is an OpenHuman abstract tier alias that resolves server-side to a concrete model. Today it resolves to DeepSeek v4 Flash (text-only) — see src/openhuman/inference/provider/config_rejection.rs:277 ("The supported API model names are deepseek-v4-pro or deepseek-v4-flash, but you passed reasoning-v1"), factory.rs:533/576, router.rs:6, model_context.rs:180. Vision is per-model, not per-tier — so an image turn on reasoning-v1 lands on a text-only model even with #3268 merged + flag on.	Backend PR — either (i) add a vision:true SKU under the reasoning-v1 resolution graph and route image-bearing turns to it when supports_vision() is required, or (ii) introduce a sibling tier (vision-v1?) the agent loop switches to when image_marker_count > 0. Note: this issue's "Quen 3.6 / 3.6+" registration is the obvious candidate for that vision-capable SKU (e.g. mark qwen-3.6-plus as vision-capable in traits.rs::supports_vision).
Chat models updated (Quen 3.6 / 3.6+)	No catalog entries today.	Register qwen-3.6 + qwen-3.6-plus in the provider catalog (likely under src/openhuman/inference/types.rs / preset module). Mark the vision-capable variant's supports_vision() return true so the gate at engine/core.rs:196-217 actually lets image turns through. Default-routing test pinning chat-v1 / reasoning-v1 resolution per config.
Default routing verified	n/a	Single regression test that pins the default chat-model resolution from a stock Config (per the routing logic in router.rs:6 + factory.rs:533) so a later refactor can't silently swap the default.
Failure messaging improved	PresentationError::UnsupportedFileType { extension, supported } scaffolded in #3029 (src/openhuman/tools/impl/presentation/types.rs) but not constructed by execute(). No structured error for the dispatch-time rejections (vision capability mismatch, MIME rejection, file-too-large).	Wire a sibling UnsupportedMime { mime, supported_list } (or extend the existing variant) and construct it at the actual rejection paths in src/openhuman/agent/multimodal.rs::parse_file_markers / parse_image_markers — not just the presentation tool. Surface the structured shape through the chat error pipeline (ChatErrorEvent in app/src/services/chatService.ts) so the React side can map error code → t(…) localized string instead of swallowing into a generic "something went wrong" (which was the symptom #3205 closed).
PDF support	Done — [FILE:…] markers + pdf_extract::extract_text_from_mem (60s timeout) in #2954.	n/a.
Capability routing safe	supports_vision() gate active (engine/core.rs:196-217). Once a vision-capable backend SKU is registered, the gate will let it through.	Implicit on backend PR.
Regression safety — image + PDF E2E	#3268 has its own coverage for the conversion path; PDF unit-covered in #2954.	Playwright spec covering the full image-attach → backend-vision-model → assistant-response flow with flag on. PDF E2E to lock in #2954 behaviour against future refactors.
Diff coverage ≥ 80%	n/a	Per-PR concern.

Revised PR sequencing

1. feat(chat): implement image attachment pipeline, gated off (#3205) #3268 → merge + flag flip (already in flight). Closes the client wire-format AC. Owner: sanil-23.
2. PR — backend model registration + vision routing. Adds Quen 3.6 / 3.6+ to the catalog, marks the vision-capable SKU as supports_vision()=true, hooks reasoning-v1 (or a new vision-v1 tier) to route image turns there. Backend-side work — the load-bearing remaining blocker. This is the critical-path PR for "Image support enabled".
3. PR — structured UnsupportedMime / UnsupportedFileType failure path. Wires the variant from real rejection paths in multimodal.rs + extends ChatErrorEvent so the React side stops surfacing generic errors. Closes "Failure messaging improved".
4. PR — image + PDF E2E coverage under the post-merge flag-on state. Closes "Regression safety".

PR #2 is the only one that requires backend coordination beyond this repo. The rest are local.

Cross-refs

• feat(chat): implement image attachment pipeline, gated off (#3205) #3268 (feat(chat): implement image attachment pipeline, gated off) — OPEN, tip f9485609. Owner: @sanil-23. Client-side wire format.
• fix(chat): hide image attachment button until backend supports it (#3205) #3212 (fix(chat): hide image attachment button until backend supports it) — MERGED 2026-06-02. Introduced CHAT_ATTACHMENTS_ENABLED flag.
• Attachments trigger Something went wrong in chat #3205 — CLOSED. Original symptom ("Attachments trigger Something went wrong in chat") that motivated fix(chat): hide image attachment button until backend supports it (#3205) #3212 + feat(chat): implement image attachment pipeline, gated off (#3205) #3268.
• Extend multimodal input to accept document and file attachments beyond images #2777 (PR feat(agent/multimodal): support [FILE:…] markers with text extraction (#2777) #2954) — MERGED. [FILE:…] markers + PDF extraction. PDF AC.
• Wire PPT generation and file attachments into agent definitions and error handling #2780 (PR feat(orchestrator): wire generate_presentation + require grounding (#2780) #3029) — OPEN. UnsupportedFileType variant scaffolding (not yet constructed).
• feat(orchestrator): wire generate_presentation + require grounding (#2780) #3029's variant is the right shape to extend for Update agent chat models and enable image/PDF multimodal support #3282's failure-messaging AC — coordinating those two would avoid a churn-y enum bump later.

Apologies for the earlier overstatement — the wire format is sanil's already, this issue's load-bearing remaining work is backend model registration + capability routing, not the client conversion.

[sanil-23]

Partial completion — the client-side image pipeline has landed but this stays open.

Done (#3268, feat(chat): implement image attachment pipeline, gated off):

• [IMAGE:<data-uri>] markers promoted to OpenAI image_url content-array parts (correct multimodal wire format).
• Token counting, context-compaction summarizer, and episodic-memory ingest all skip the image base64.
• Composer attach UI present but gated behind CHAT_ATTACHMENTS_ENABLED (default off, from fix(chat): hide image attachment button until backend supports it (#3205) #3212).

Remaining scope to close #3282:

• Enable CHAT_ATTACHMENTS_ENABLED once the backend is ready.
• Route image turns to a vision-capable model (agent chat model update).
• PDF multimodal support (not covered by feat(chat): implement image attachment pipeline, gated off (#3205) #3268 — images only).