Mutimodule - #288
Open
rhinewg wants to merge 7 commits into
Open
Conversation
…nd providers - Introduce ContentPart-based multimodal schema and context propagation. - Add OpenAI, Anthropic, and Gemini adapters for image_url content parts. - Enable microservice JSON + multipart image uploads (data_url default; store mode gated by AGENT_SDK_UPLOAD_DIR) with URL/mime validation. - Extend agent-cli with --image/--image-url flags and :image chat attachments.
- Allow UI /agent/run and /agent/stream to accept content_parts (input or parts required) - Add image picker in Next.js UI and send images as data URLs via content_parts - Update UI request types and refresh embedded static UI build output
- Replace native file input with button-triggered hidden input for consistent layout. - Add optional ChatMessage.display_content for UI-friendly rendering. - Fix UI lint blockers (effects/state, remove explicit any casts). - Regenerate embedded Next.js out/ assets. - Add ui_multimodal_server example docs and starter.
golangci-lint (staticcheck): - ST1005: error strings lowercase, no trailing punctuation (cmd/agent-cli/multimodal.go, pkg/microservice/http_server.go) - S1030: use getW.Body.String() in http_server_test.go gosec: - G304: restrict file read to cwd via os.OpenRoot (multimodal.go) - G301: upload dir permissions 0o755 -> 0o750 (http_server.go) - G306: uploaded file permissions 0o644 -> 0o600 (http_server.go)
…into mutimodule
|
|
||
| ### 5. UI Multimodal Server (`ui_multimodal_server/`) | ||
|
|
||
| 演示如何启动带 **内置 UI** 的 HTTP 服务（`HTTPServerWithUI`），并包含一个 `client` 使用用例：从命令行读取图片并以 `content_parts` 调用 `/api/v1/agent/run` 验证多模态链路。 |
Contributor
There was a problem hiding this comment.
Please, could you translate this text to English?
|
@meidad That PR seems to haven't moved in more than 2 months and seems incomplete with things like front end artifacts. The user seems to be very sporadically on GitHub without any activity since February. Mind if I pick it up and finalize it? |
Collaborator
|
Solid feature — image inputs across CLI, microservice, and all three LLM providers (OpenAI, Anthropic, Gemini) in one PR. The security attention (G304/G301/G306) is appreciated. One blocker and one question: Required
Questions
Nice-to-have
The core multimodal logic looks clean. Happy to merge once the compiled assets are removed. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
[FEATURE] Support multimodal input #272
Description
This PR adds multimodal input support so users can send not only text but also images when interacting with agents. The library previously only supported text input; this change extends the CLI, HTTP microservice server, and optional Web UI to accept image URLs, local image files, and multipart file uploads.
Summary of changes:
content_parts(e.g.textandimage_urlparts) in addition to the existinginputtext field. Image URLs may behttp://,https://, ordata:(base64) and are validated for scheme and MIME type where applicable.--image-urland--image-pathto attach images (URLs or local paths). Local file reads are scoped to the current directory viaos.OpenRoot(Go 1.24) to prevent path traversal (gosec G304).pkg/microservice):content_partsin the request body.images/imagefile fields, optionalupload_mode(dataURL vsstore), and optionalcontent_partsJSON field.HTTPServerWithUI): Chat UI supports attaching images and sending them ascontent_parts(e.g. data URL or stored file URL). Display content is separated from payload content where needed for correct rendering.examples/microservices/ui_multimodal_serverdemonstrating the UI and multimodal API.Voice input is out of scope for this PR and can be addressed in a follow-up (e.g. speech-to-text then text input, or first-class audio parts if/when the project supports it).
Fixes #272
Type of change
Please delete options that are not relevant.
How Has This Been Tested?
go build ./...and buildingcmd/agent-cliandexamples/microservices/ui_multimodal_server.golangci-lint run ./cmd/agent-cli/... ./pkg/microservice/...(0 issues); gosec run on the same paths (G304, G301, G306 resolved).go test ./pkg/microservice/...(includinghttp_server_test.gofor multipart and file handling).--image-url/--image-path.content_partswithimage_url.imagesfile upload.http://localhost:8085/(or configured port), attach image, send message and confirm request usescontent_parts.Reproduce: run
go run ./examples/microservices/ui_multimodal_server, open the UI URL, attach an image and send a prompt; optionally call the HTTP API withcontent_partsor multipart from another client.Checklist: