feat(ai-openai): PDF document input

[dangreenberggit]

feat(ai-openai): support PDF document content parts in the Responses adapter

🎯 Changes

openaiText's Responses adapter now accepts PDF document content parts. Any OpenAI chat model whose model-meta entry declares the document input modality can take a PDF as message content.

• Base64 (data) sources send input_file with a file_data data URL and a filename (from metadata.filename, defaulting to document.pdf).
• URL sources send input_file with file_url.
• Non-PDF MIME types are rejected before the request goes out — including pre-wrapped data: URLs whose media type doesn't match the declared mimeType — so callers get a clear error instead of an opaque provider 400.
• OpenAIDocumentMetadata gains filename and detail.
• Documents are Responses-only; the Chat Completions adapter throws a document-specific error naming the Responses path.

import { openaiText } from '@tanstack/ai-openai/adapters'

const adapter = openaiText('gpt-5.5')

const message = {
  role: 'user',
  content: [
    { type: 'text', content: 'Summarize this document' },
    {
      type: 'document',
      source: { type: 'data', value: pdfBase64, mimeType: 'application/pdf' },
      metadata: { filename: 'report.pdf' },
    },
  ],
}

Test plan

• Unit: Responses adapter mapping for base64 and URL sources, filename defaulting, and the MIME guards (non-PDF mimeType, mismatched data: URL media type); the Chat Completions document-rejection error; model-meta modality assertions.
• E2E: new multimodal-document feature — fixture, spec (testing/e2e/tests/multimodal-document.spec.ts), harness page, and a tiny.pdf asset (a minimal PDF), wired into feature-support.ts and features.ts.
• Docs: document example in docs/advanced/multimodal-content.md; the modality list points readers at model-meta.ts.

✅ Checklist

• I have followed the steps in the Contributing guide.
• I have tested this code locally with pnpm run test:pr.

🚀 Release Impact

• This change affects published code, and I have generated a changeset.
• This change is docs/CI/dev-only (no release).

Summary by CodeRabbit

• New Features
  • Added PDF document (“document” modality) support for compatible OpenAI Responses models (base64 or URL sources), including optional filename and detail, with sensible defaults like document.pdf.
  • Enabled PDF attachments in the multimodal chat UI and added a new multimodal-document end-to-end test and fixture.
• Bug Fixes
  • Improved validation and error messaging for invalid/non-PDF inputs; document parts are now routed correctly for Responses (Chat Completions rejects them).
• Documentation
  • Updated OpenAI provider multimodal docs and the supported-modality list to include PDF documents.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

OpenAI model metadata now declares PDF document input support. The Responses adapter converts PDF data and URL sources into input_file payloads, while Chat Completions rejects documents. Documentation, E2E upload support, fixtures, and validation tests are added.

Changes

OpenAI PDF document support

Layer / File(s)	Summary
Document modality contracts packages/ai-openai/src/message-types.ts, packages/ai-openai/src/model-meta.ts, packages/ai-openai/tests/model-meta.test.ts	Document metadata gains filename and detail, and supported OpenAI models now allow document input.
Responses document conversion packages/openai-base/src/adapters/responses-text.ts, packages/openai-base/tests/responses-text.test.ts	PDF data and URL sources are converted to input_file payloads with MIME and base64 validation, metadata forwarding, ordering, and tool-result coverage.
Chat Completions document rejection packages/openai-base/src/adapters/chat-completions-text.ts, packages/openai-base/tests/chat-completions-text.test.ts	Chat Completions rejects document parts with a Responses adapter error before making an SDK request.
Document upload E2E flow testing/e2e/src/*, testing/e2e/tests/*, testing/e2e/fixtures/*, testing/e2e/README.md	The multimodal-document feature adds PDF upload UI wiring, provider support configuration, upload helpers, fixtures, and an end-to-end test.
Release and documentation updates .changeset/*, docs/advanced/*, docs/config.json	Package minor releases and PDF document examples, model support details, and update metadata are recorded.

Estimated code review effort: 3 (Moderate) | ~30 minutes

Suggested reviewers: tombeckenham

Sequence Diagram(s)

sequenceDiagram
  participant ChatUI
  participant ChatFeature
  participant ResponsesAdapter
  participant OpenAIResponsesAPI
  ChatUI->>ChatFeature: select PDF and submit prompt
  ChatFeature->>ChatFeature: read PDF as base64 data URL
  ChatFeature->>ResponsesAdapter: send document content part
  ResponsesAdapter->>ResponsesAdapter: validate PDF and build input_file
  ResponsesAdapter->>OpenAIResponsesAPI: submit file_data or file_url
  OpenAIResponsesAPI-->>ChatFeature: stream response events

Loading

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 28.57% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title is concise and accurately summarizes the main change: PDF document input support for the OpenAI adapter.
Description check	✅ Passed	The description matches the template with Changes, Test plan, Checklist, and Release Impact sections, and includes the required implementation details.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/ai-openai/src/message-types.ts`:
- Around line 62-66: Allow OpenAIDocumentMetadata.detail to include 'auto' and
update its documentation to state that the API defaults to auto, resolving to
low or high by model; in packages/ai-openai/src/message-types.ts lines 62-66,
widen the type and revise the comment. Also widen the inline detail cast in
packages/openai-base/src/adapters/responses-text.ts lines 1843-1845 to accept
'auto' alongside the existing values.

In `@packages/openai-base/src/adapters/responses-text.ts`:
- Around line 1864-1873: Update the document handling logic in the relevant
responses-text conversion method so raw base64 documents without a mimeType
undergo the same local PDF validation instead of defaulting directly to
application/pdf and bypassing checks. Preserve explicit mimeType normalization
and rejection of non-PDF values, and add a regression test covering a raw base64
document with no mimeType.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 39b1fe3a-2bad-4186-b445-34af67ff1dc0

📥 Commits

Reviewing files that changed from the base of the PR and between bcb1b93 and b20a655.

⛔ Files ignored due to path filters (1)

• testing/e2e/test-assets/tiny.pdf is excluded by !**/*.pdf

📒 Files selected for processing (19)

• .changeset/openai-pdf-document-input.md
• docs/advanced/multimodal-content.md
• docs/config.json
• packages/ai-openai/src/message-types.ts
• packages/ai-openai/src/model-meta.ts
• packages/ai-openai/tests/model-meta.test.ts
• packages/openai-base/src/adapters/chat-completions-text.ts
• packages/openai-base/src/adapters/responses-text.ts
• packages/openai-base/tests/chat-completions-text.test.ts
• packages/openai-base/tests/responses-text.test.ts
• testing/e2e/README.md
• testing/e2e/fixtures/multimodal-document/basic.json
• testing/e2e/src/components/ChatUI.tsx
• testing/e2e/src/lib/feature-support.ts
• testing/e2e/src/lib/features.ts
• testing/e2e/src/lib/types.ts
• testing/e2e/src/routes/$provider/$feature.tsx
• testing/e2e/tests/helpers.ts
• testing/e2e/tests/multimodal-document.spec.ts

[coderabbitai]

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

testing/e2e/src/components/ChatUI.tsx (1)

268-284: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Reset the file input after handling the document.

Clearing input does not clear the native file input’s value, so selecting the same PDF again will not trigger onChange. Reset the file input element after invoking the callback, e.g. event.currentTarget.value = ''.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@testing/e2e/src/components/ChatUI.tsx` around lines 268 - 284, Update the
document file input’s onChange handler in ChatUI so that after invoking
onSendMessageWithDocument, it also clears the native file input via the event
target value; preserve the existing text-input reset and callback conditions.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@testing/e2e/src/components/ChatUI.tsx`:
- Around line 268-284: Update the document file input’s onChange handler in
ChatUI so that after invoking onSendMessageWithDocument, it also clears the
native file input via the event target value; preserve the existing text-input
reset and callback conditions.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: bfc978c7-da8d-4bb5-9125-459129ae4af2

📥 Commits

Reviewing files that changed from the base of the PR and between 0f81a09 and e5d210e.

📒 Files selected for processing (6)

• .changeset/openai-pdf-document-input.md
• packages/ai-openai/src/message-types.ts
• packages/openai-base/src/adapters/responses-text.ts
• packages/openai-base/tests/responses-text.test.ts
• testing/e2e/src/components/ChatUI.tsx
• testing/e2e/tests/helpers.ts

🚧 Files skipped from review as they are similar to previous changes (5)

• packages/ai-openai/src/message-types.ts
• testing/e2e/tests/helpers.ts
• .changeset/openai-pdf-document-input.md
• packages/openai-base/src/adapters/responses-text.ts
• packages/openai-base/tests/responses-text.test.ts