feat(inpainting): add inpainting endpoint, wire ImageGenerationFunc and return generated image URL #7328
Conversation
✅ Deploy Preview for localai ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
gmaOCR
force-pushed
the
fix/inpainting-backend-response-and-cuda
branch
from
1a714e7 to
68bfe00
Compare
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| // write JSON | ||
| enc := json.NewEncoder(jf) | ||
| if err := enc.Encode(jsonFile); err != nil { | ||
| jf.Close() |
Check warning
Code scanning / gosec
Errors unhandled
| enc := json.NewEncoder(jf) | ||
| if err := enc.Encode(jsonFile); err != nil { | ||
| jf.Close() | ||
| os.Remove(jf.Name()) |
Check warning
Code scanning / gosec
Errors unhandled
| os.Remove(jf.Name()) | ||
| return err | ||
| } | ||
| jf.Close() |
Check warning
Code scanning / gosec
Errors unhandled
| jf.Close() | ||
| // rename to desired name | ||
| if err := os.Rename(jf.Name(), jsonPath); err != nil { | ||
| os.Remove(jf.Name()) |
Check warning
Code scanning / gosec
Errors unhandled
| // prepare dst | ||
| outTmp, err := os.CreateTemp(tmpDir, "out_") | ||
| if err != nil { | ||
| os.Remove(jsonPath) |
Check warning
Code scanning / gosec
Errors unhandled
| os.Remove(jsonPath) | ||
| return err | ||
| } | ||
| outTmp.Close() |
Check warning
Code scanning / gosec
Errors unhandled
| outTmp.Close() | ||
| dst := outTmp.Name() + ".png" | ||
| if err := os.Rename(outTmp.Name(), dst); err != nil { | ||
| os.Remove(jsonPath) |
Check warning
Code scanning / gosec
Errors unhandled
| // Note: ImageGenerationFunc will call into the loaded model's GenerateImage which expects src JSON | ||
| fn, err := backend.ImageGenerationFunc(height, width, 0, steps, 0, prompt, "", jsonPath, dst, ml, *cfg, appConfig, nil) | ||
| if err != nil { | ||
| os.Remove(jsonPath) |
Check warning
Code scanning / gosec
Errors unhandled
|
|
||
| // Execute generation function (blocking) | ||
| if err := fn(); err != nil { | ||
| os.Remove(jsonPath) |
Check warning
Code scanning / gosec
Errors unhandled
| // Execute generation function (blocking) | ||
| if err := fn(); err != nil { | ||
| os.Remove(jsonPath) | ||
| os.Remove(dst) |
Check warning
Code scanning / gosec
Errors unhandled
gmaOCR
force-pushed
the
fix/inpainting-backend-response-and-cuda
branch
from
68bfe00 to
1d33fbf
Compare
github-actions
bot
added
the
kind/documentation
gmaOCR
force-pushed
the
fix/inpainting-backend-response-and-cuda
branch
from
1d33fbf to
5884e9c
Compare
|
|
||
| // Call backend image generation via indirection so tests can stub it | ||
| // Note: ImageGenerationFunc will call into the loaded model's GenerateImage which expects src JSON | ||
| fn, err := backend.ImageGenerationFunc(height, width, 0, steps, 0, prompt, "", jsonPath, dst, ml, *cfg, appConfig, nil) |
There was a problem hiding this comment.
ImageGeneration actually take a list of refimages files. We could wire these up to the image and mask, so it becomes open to the backend to implement this correctly.
There was a problem hiding this comment.
Hi thanks for the pointer, good catch.
I wired the uploaded image and mask as reference image files and pass them to the backend via the refImages parameter so backends that expect file references can use them directly. For compatibility I kept the existing JSON src (base64) as well, and added robust best‑effort cleanup of all temp files. The inpainting endpoint now does:
- write image and mask to temp files (origRef, maskRef)
- write the src JSON (base64) as before (for backward compatibility)
- call backend.ImageGenerationFunc(..., refImages) with both paths
mudler
added
enhancements
and removed
kind/documentation
gmaOCR
force-pushed
the
fix/inpainting-backend-response-and-cuda
branch
from
7e414a1 to
8aa0b43
Compare
- Created gallery/dreamshaper-8-inpainting.yaml with diffusers backend - Configured StableDiffusionInpaintPipeline for inpainting tasks - Added gallery index entry with model file mapping - Maps 'dreamshaper-8-inpainting' API name to DreamShaper_8_pruned.safetensors file - Fixes CI test failures: model file resolution now works via gallery system - Addresses maintainer feedback on PR mudler#7328 Signed-off-by: Greg <[email protected]>
gmaOCR
force-pushed
the
fix/inpainting-backend-response-and-cuda
branch
2 times, most recently
from
80af855 to
8bfe58a
Compare
github-actions
bot
added
the
kind/documentation
gmaOCR
force-pushed
the
fix/inpainting-backend-response-and-cuda
branch
9 times, most recently
from
d3d55f2 to
5d1cf58
Compare
- Add inpainting endpoint supporting OpenAI-compatible API - Implement automatic model selection for image generation endpoints - Add comprehensive tests for inpainting functionality - Update Swagger documentation for new endpoint - Wire ImageGenerationFunc to backend Signed-off-by: Greg <[email protected]> ci: re-trigger tests-apple workflow
gmaOCR
force-pushed
the
fix/inpainting-backend-response-and-cuda
branch
from
79941c8 to
874b052
Compare
2 participants
Description
This patch implements an OpenAI-compatible inpainting endpoint and wires image generation into the LocalAI backend. Changes are limited to the LocalAI Go codebase.
Adds InpaintingEndpoint handling POST /v1/images/inpainting with multipart/form-data support for image and mask.
Endpoint writes a JSON payload with base64 image/mask and prepares a temporary output PNG path, then calls backend.ImageGenerationFunc(...) to perform generation.
Introduces and uses the test-friendly indirection ImageGenerationFunc in image.go so the generation logic can be stubbed in tests.
Adds unit tests for the inpainting handler (inpainting_test.go) covering missing-files and the happy path (with backend.ImageGenerationFunc stubbed).
Registers the inpainting routes and a default model name middleware in openai.go.
Updates Swagger (swagger.yaml) and adds a Swaggo-style docstring on the new endpoint for proper OpenAPI generation.
Notes for Reviewers
Signed commits