Add optional PDF and image inputs to extract.ai

[ebhills]

Outcome

Allow ordinary extract.ai calls and YAML recipes to extract structured information directly from PDFs and images, optionally combined with text. This should make diagrams, photographs, scanned text, and document layout available to a capable model without requiring callers to convert everything to Markdown first.

The current Responses adapter in wrangles/openai_responses.py constructs a text-only user message from format_input_data(data). Selecting a vision-capable model alone therefore does not supply the original document or image.

Goals and acceptance criteria

• Explicit attachment inputs. Define a small, clear public contract for one or more PDF/image attachments per input record, with optional text context. Support PDF-only, image-only, and mixed requests. Local files are an initial use case; agree on the other supported forms, such as bytes, URLs, or provider file IDs, during design. Ordinary text containing a path or URL must not silently trigger loading or uploading.
• Preserve compatibility. Keep existing text-only behavior, structured-output schemas, saved models, configuration precedence, and return shapes. A Python input list currently represents separate extractions; do not reinterpret that list as attachments to one extraction. Recipe input-column selection, row association, and output order must remain predictable.
• Use the existing wrangle. Expose this through the Python API and normal YAML extract.ai configuration. Recipes continue to control discovery, mapping, checkpoints, and subsequent stages. Callers should not need a parallel custom API client.
• Send actual multimodal content. Start with the OpenAI Responses path and caller-selected supported models. Validate missing files, unsupported formats, size limits, and incompatible model/provider/protocol combinations with actionable errors. Never silently discard visual inputs. Document any upload lifecycle introduced. Hosted PDF/image input should not require Docling or a local GPU.
• Retain operational behavior. Integrate with existing credential resolution, including model/group-scoped credentials, retries, concurrency, request storage, metadata, and logging. Cache identity must include attachment content and relevant request settings: replacing a file at the same path must invalidate the result. Distinguish attachment ordering and association with records.
• Account for usage. Preserve provider-reported input/output tokens and available cached-input, cache-write, and reasoning breakdowns, along with model identity, request status, timing, and request/response identifiers. Include failed or incomplete attempts when usage is returned. Do not double-count reasoning already included in output tokens; missing usage is unknown, not zero. Diagnostics should distinguish local cache reuse from a new model call and enable downstream cost estimates without hard-coding price tables into the wrangle. Normal logs must not expose credentials or full binary/Base64 payloads.
• Enable source references. Preserve stable file/image identifiers and make them usable in caller-defined output schemas and prompts requesting page numbers, quotes, or image references. These remain model-produced claims requiring validation. This first slice does not promise accurate bounding boxes or table-cell coordinates. Native PDF/Docling location matching, known crop coordinates, and highlight generation can be added downstream.

Verification and documentation

• Add small synthetic or sanitized fixtures and focused tests covering text-only regressions, PDF/image request construction, mixed and multiple attachments, row/list semantics, source association, validation failures, attachment cache identity, credential isolation, and incomplete-response usage.
• Provide simple Python and YAML recipe examples, including text plus an attachment and multiple input records. Document the supported models/protocols, input forms, limits, and diagnostics.
• Report offline tests separately from live-service validation. Validate the supported paths with an in-scope PDF trial, a standalone image, and mixed text/image input; retain actual per-attempt usage and timings.
• Use the RSGroup SF_AMF60 case as a downstream integration check. Review remaining quality gaps explicitly; the first usable slice may have documented extraction errors.

Evidence motivating this work

A local RSGroup experiment supplied the original SF_AMF60 PDF to Astra at medium reasoning, then performed a separate mapping call. Against 30 frozen source checks, the PDF workflow achieved 27 pass, 2 partial, 1 absent, compared with 15 pass, 7 partial, 8 absent for the existing Docling/Granite-to-text workflow using Luna. Both found four product variants.

The successful PDF discovery and mapping calls used 54,056 input and 30,133 output tokens, estimated at $2.18 using the experiment's September 10, 2026 rates. Including an initial incomplete attempt, the experiment cost approximately $3.62. Model, prompt, workflow, and output budget differed, so this is encouraging evidence rather than a controlled modality-only comparison.

Local, currently uncommitted RSGroup references: run_pdf_vision.py, functions_pdf_vision.py, recipes/pdf_vision.recipe, and pipeline/extraction/pdf_vision_api.py. The local report and accounting are under .data/pdf-vision/SF_AMF60/091026.02/. Treat this prototype as design evidence, not the required public API.

Scope and handoff

This issue concerns reusable WranglesPY inputs. Product discovery schemas, customer attribute mapping and overrides, complete per-product rows, mapping-validation fixes, Excel presentation, and Docling provenance resolution remain RSGroup responsibilities. Deployment and Excel UI exposure require separate follow-up where needed.

Eric will start a WranglesPY task from this issue. Resume RSGroup integration once the capability is available.

Related diagnostics work: #1172. Build on existing logging; an optional tracing exporter is not a prerequisite.

Provider references: PDF inputs, Images and vision.