Skip to content

Add image refinement flow to our image generation screens#292

Open
dkotter wants to merge 19 commits intoWordPress:developfrom
dkotter:feature/image-edits
Open

Add image refinement flow to our image generation screens#292
dkotter wants to merge 19 commits intoWordPress:developfrom
dkotter:feature/image-edits

Conversation

@dkotter
Copy link
Collaborator

@dkotter dkotter commented Mar 10, 2026

What?

Closes #279

Partially closes #270

Add a "Refine" flow to our existing image generation screens

Why?

It's likely that an image you generate doesn't match exactly what you want the first time. While you can edit the prompt and try again, sometimes the image generated is pretty close and needs just a few refinements. This flow is now supported when generating images, both in the editor and directly in the Media Library.

How?

  • Update our generate-image Ability to support passing in a reference base-64 encoded image. If that is passed, we add that as a file to our prompt
  • In our image generation UI, add a "refine" state that allows you to enter a refinement prompt and send that along with the generated image (this was previously added in Inline image generation #235 but removed)
  • When a refinement comes back, show the original image next to the refined image for easy comparison
  • When importing an image, keep track of each prompt used so we can store that with the image. Previously we stored the original prompt only, now we store that plus any refinement prompts

Use of AI Tools

Pulled my original code from #235 for most of this. Used GPT-5.3 Codex in Cursor to refine and fix a few things with manual review, testing and cleanup done by me

Testing Instructions

  1. Install the AI Provider for Google plugin
  2. Pull this PR down and run npm install && npm run build
  3. Add valid Google credentials
  4. Turn on the Image Generation experiment
  5. Add an image block into a post and click on the Generate Image button
  6. Add a prompt to generate an image
  7. Click the Refine Image button and add another prompt
  8. Choose to keep that image, refine again or generate again (which will send the original image along with the refinement prompt, basically a "try again" approach)
  9. Ensure all those buttons work as expected and the image imports with proper history
  10. Do the same thing with the stand-alone image generation screen at Media > Generate Image

Screenshots

Modal showing how to generate a new image Generated image showing available options, including the new Refine image button Adding a refinement prompt to a generated image Showing the result of the refinement step with the original image next to the refined image Open WordPress Playground Preview

dkotter added 7 commits March 9, 2026 16:10
…n passed, add that as a reference image before making our image generation request
…mage for easier comparison. In that state, ensure refining further refines the refined image instead of the original. And ensure generating another image sends the original image along with the refinement prompt
@dkotter dkotter added this to the 0.5.0 milestone Mar 10, 2026
@dkotter dkotter self-assigned this Mar 10, 2026
@codecov
Copy link

codecov bot commented Mar 10, 2026

Codecov Report

❌ Patch coverage is 83.33333% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 57.43%. Comparing base (71f393a) to head (32adbf2).

Files with missing lines Patch % Lines
includes/Abilities/Image/Generate_Image.php 82.14% 5 Missing ⚠️
Additional details and impacted files
@@              Coverage Diff              @@
##             develop     #292      +/-   ##
=============================================
+ Coverage      57.05%   57.43%   +0.37%     
- Complexity       591      596       +5     
=============================================
  Files             42       42              
  Lines           3083     3101      +18     
=============================================
+ Hits            1759     1781      +22     
+ Misses          1324     1320       -4     
Flag Coverage Δ
unit 57.43% <83.33%> (+0.37%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@dkotter dkotter marked this pull request as ready for review March 16, 2026 15:33
@github-actions
Copy link

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Co-authored-by: dkotter <dkotter@git.wordpress.org>
Co-authored-by: jeffpaul <jeffpaul@git.wordpress.org>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

@dkotter dkotter requested a review from jeffpaul March 16, 2026 15:43
@jeffpaul
Copy link
Member

In the resulting image compare modal post-refinement, its not immediately clear which image relates to which button

Screenshot 2026-03-16 at 4 09 45 PM

I'd say either we present ONLY the Refined image result or if we're going to show both original and refined images then we likely need to be more precise in button text (e.g. Use refined image) or have different buttons aligned under each image option (e.g. Use original image and Use refined image).

Continuing to think through this... given someone wanted to refine an image, I'm not immediately concerned with presenting them the original though perhaps post-refinement there's still a Cancel Refinement option that takes them back to the previously generated image and otherwise display the Refined image image with the other buttons as already exist?

@jeffpaul
Copy link
Member

Also, probably need to adjust the Image Generation experiment title to something like Image Generation and Editing with similar update to description.

Screenshot 2026-03-16 at 4 16 55 PM

@dkotter
Copy link
Collaborator Author

dkotter commented Mar 17, 2026

I'd say either we present ONLY the Refined image result or if we're going to show both original and refined images then we likely need to be more precise in button text (e.g. Use refined image) or have different buttons aligned under each image option (e.g. Use original image and Use refined image).

Continuing to think through this... given someone wanted to refine an image, I'm not immediately concerned with presenting them the original though perhaps post-refinement there's still a Cancel Refinement option that takes them back to the previously generated image and otherwise display the Refined image image with the other buttons as already exist?

So the goal here is to make it easy for them to compare and see what changes have happened. I personally think that's nice, especially if you refine an image multiple times.

Worth noting I do have a better approach for this I bundled in the next PR (#305). In there, we highlight the "active" image and also keep track of each refined image, so you can paginate back and forth (see the screenshots in the PR description).

So in that approach, if you refine an image 5 times, each time we show the image that was used for the refinement on the left and the newly generated image on the right, highlighting the active image. You can them import that image or toggle back to another image (say the 3rd image you generated) and then import that instead.

But I can change all of that to show a single image at a time if we think that's the better approach.

@jeffpaul jeffpaul mentioned this pull request Mar 18, 2026
32 tasks
@jeffpaul
Copy link
Member

Ok yeah I see what you mean in #305, that's probably a solid iteration to fold back into the work here (or extend that one to include the other work in this PR, whichever you prefer). I've got some feedback for 305 that I'll jump over and leave there now.

@dkotter
Copy link
Collaborator Author

dkotter commented Mar 18, 2026

Ok yeah I see what you mean in #305, that's probably a solid iteration to fold back into the work here (or extend that one to include the other work in this PR, whichever you prefer). I've got some feedback for 305 that I'll jump over and leave there now.

So #305 was branched from this PR so the changes in this PR are already applied to that PR (and vice versa I guess, the UI improvements made there also impact here). Goal was to get this one merged first and then can further iterate on that PR as needed

@jeffpaul
Copy link
Member

Let's also move the Image Generation and Editing settings page item out of the Editor Experiments section into a new section above. Let's call that entire section Image Generation and Editing with the same description text and a placeholder for a descriptive image (300x250?). I'll spend a little time looking to generate an image for that slot. I'm not sure how best to signal there that its a Feature and not an Experiment, so I'm open to ideas (feel free to just YOLO into whatever strikes you).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Normalize Media Library image generation to editor flow Refine image generation loading state and fix layout shift in Generate Image modal

2 participants