Outpainting with Diffusers

[asomoza]

Hi, I have made a 3 part series about outpainting with Diffusers, each one with a different method.

Outpainting I - Controlnet version

Outpainting using a controlnet trained for this. This method does change the original image but has the advantage that the ourpaining its more creative and seamless.

Outpainting II - Differential Diffusion

This method uses differential diffusion which allows us to use a soft inpainting mask that generates seamless outpainting, as the first method I can change the original image but not as much.

Outpainting III - Inpaint Model

The final method uses an inpainting model but it will preserve the original image so it's more suitable for professional work.

[tolgacangoz]

[asomoza]

fixed, thanks.

[crapthings]

control_image = Image.open(BytesIO(response.content))
new_controlnet_image = Image.new("RGBA", control_image.size, "WHITE")
new_controlnet_image.alpha_composite(control_image)

why this dones't work?

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[11], line 17
     14 b = midas(input_image)
     16 new_controlnet_image = Image.new('RGB', input_image.size, 'WHITE')
---> 17 new_controlnet_image.alpha_composite(ccc)
     18 # new_controlnet_image = Image.composite(input_image, new_controlnet_image, input_image)
     20 display(new_controlnet_image)

File [~/.local/lib/python3.10/site-packages/PIL/Image.py:1790](http://192.168.50.251:8888/lab/tree/workspace/ai-pipe/ai-pipe-replace-model-new/.local/lib/python3.10/site-packages/PIL/Image.py#line=1789), in Image.alpha_composite(self, im, dest, source)
   1787 else:
   1788     background = self.crop(box)
-> 1790 result = alpha_composite(background, overlay)
   1791 self.paste(result, box)

File [~/.local/lib/python3.10/site-packages/PIL/Image.py:3328](http://192.168.50.251:8888/lab/tree/workspace/ai-pipe/ai-pipe-replace-model-new/.local/lib/python3.10/site-packages/PIL/Image.py#line=3327), in alpha_composite(im1, im2)
   3326 im1.load()
   3327 im2.load()
-> 3328 return im1._new(core.alpha_composite(im1.im, im2.im))

ValueError: image has wrong mode

[asomoza]

When you're doing alpha composite you need both images to have an alpha channel. Probably the image you're loading as a control_image is a RGB image instead of a RGBA

[juzstu]

I find that the BrushNet also performs well in outpainting.

[asomoza]

Thanks, I have it under my radar, I'm waiting for the SDXL version but it seems that's going to take more time.

[juzstu]

@asomoza For your Outpainting III - Inpaint Model, I use lora to finetune the RealVisXL_V4.0, but the image quality was really bad, if not use the lora weights, the image quality turn to normal, can you tell me what case this and how can I use lora weights correctly to generate a normal quality image?

[asomoza]

That is a step by step guide, I provide a full script at the end but it's just so people can learn and follow it.

If you're just going to use it and change something, you should divide it into two parts, one for generating the temp background and the second part to actually generate the background with an inpainting model. Also you'll need to remove the fixed seeds from the code.

I'll need a little more information on what you're trying to do, are you loading the LoRA in the first step? the second? both? For what are you using the LoRA? the LoRA works ok if you just use it with a normal generation?

For example, if I just want to change the style of the generated background, I can just load the pixel art LoRA on the second step, and change the prompt for that one with the trigger for the LoRA.

source image	bg removed

temp background	final background	pixel art backgound

Hope the images explain the process a little better, but for me to be able to help you, I'll need more information on what you're trying to do.

[juzstu]

Thanks for your replay. I use the my own data to lora finetune the RealVisXL_V4.0 and use it in the first step, if I drop the zoe-depth model from the controlnets, the image will be normal. As you mentioned that the inpaint model likes to add details, but if I use zoe-depth or canny/depth mdel, the background only has pure color and no or just little details.

[asomoza]

In my last example I changed the source image, the prompts, the seeds and added a LoRA and it worked just fine, the controlnet works as intended even if you change those.

So the problem lies in one or a couple of what I mentioned, I'm guessing you're not sharing those because you can't, so I really can't help you anymore with the information you can give me.

My last suggestion is that you can try different settings for the depth controlnet, try lowering the controlnet_conditioning_scale or the control_guidance_end, maybe your LoRA needs more freedom to work.

[juzstu]

Thank you for your suggestion, I will give it a try.

[peki12345]

@asomoza I have carefully read these three plans and have been greatly inspired. Thank you very much.
However, I have to mention an important issue: when I only have text2img model, some lora or controlnet, and can't accept changes in no-mask areas, I can only use plan 2.
But after trying plan 2, I found that it could hardly complete the structure of the human body (object), let alone the face. Moreover, when the outpainting area is too large, the quality of generated content is low, and there may be problems connecting with no-mask areas.

For example, I used Plan 2 to double the repainting of this image, but the results I obtained were not very satisfactory.

But the result of Fooocus is always good

[madstuntman11]

Still have some transition artifacts, but it works for my task. Maybe the next step is to add ip-adapter with style block active.

[peki12345]

You're comparing big images with really small ones, if you want to achieve the same results as fooocus, you can scale down the image so SDXL can generate the background in one pass, that's what fooocus is doing. If you do that, you can use the third method with the inpaint model.

@peki12345 Tried it. Works good for my case (not photo realistic, but artistic). Just do not pad with simple white or black, fill with something reasonable. Model from the 3rd example will keep it white of black it this case. For example, use random edge tiles from original image.

This sounds reasonable, I will try it. But I don't know exactly how to do it yet. Can you provide a simple code?

[peki12345]

I was going to start working on this, but what always impresses me is how people don’t look closely at the fooocus results and think it’s really good when it isn’t.

Now that I look closer to your image, I can clearly see the seams and a lot of defects, changes in skin tones, patterns, it really makes the image not usable in any kind of serious work.

on the other side, the ones you generated with diffusers don't have the seams or the clear defects, it's a smooth transition but it doesn't do the person generation. Did you try with a prompt?

I will do the guide because I really think it's outdated even if it's a couple of months old, but there was a lot of improvements in this field. Still I think trying to achieve fooocus results is no longer a high standard, I think I can get better results with just inpainting in ComfyUI right now.

prompt

Yes, I must admit that fooocus has many defects. However, the seam appears abrupt because I directly pasted the original image without merge.

Regarding the issue of prompt, both fooocus and diffusers used a simple prompt that is independent of the image content. I believe a more detailed prompt will enhance the effect, but I want to use one prompt to infer all the images. I don't intend to create an interactive interface like fooocus, which is why I chose diffusers.

[asomoza]

but I want to use one prompt to infer all the images

I can tell you with that it will fail with all the complex images if you want to use a simple prompt for all the images. What you can do is to use a VLLM to generate the prompt without user intervention, but those 3 methods I posted will need a good prompt for complex images.

I'll try to achieve what you want in the new version but I can't be sure it will work, I have some ideas on how to do it.

Also, with this:

I directly pasted the original image without merge

It's been a while since I used fooocus, you mean inside fooocus or using a external software? I remember that fooocus didn't have that many options, you use the image and tell it to expand it or use a mask on a smaller image, all of my tests that I did with those options produced horrible results (similar to the images you posted) so I just assumed I was just bad at using it.

[peki12345]

It's been a while since I used fooocus, you mean inside fooocus or using a external software? I remember that fooocus didn't have that many options, you use the image and tell it to expand it or use a mask on a smaller image, all of my tests that I did with those options produced horrible results (similar to the images you posted) so I just assumed I was just bad at using it.

Using VLLM is indeed a good idea, but I need to check if my GPU memory is sufficient.
fooocus is only the main part, there are also some pre-processing and post-processing. In pre-processing, I used some tricks to make the obtained results more stable. If you use the original fooocus, there is a certain probability that you will receive horrible results, which is not your problem.

[xduzhangjiayu]

Hey, is there any training-free method for out-painting?

[asomoza]

Hi, I did a guide for image fill, I commented at the end that you can use the same technique for out painting.

fffiloni which is very talentend and does some of the best spaces, took this knowledge and the space I made for it and implemented outpainting, you can check it here.

Even if this is not what I consider the final and the best I can do, a lot of people commented that this is the best outpainting they've seen and it's really fast. That space has a lot of cool additions too.

Also what do you mean by training-free? Even if the ones here are old and outdated, they were training-free, just inference with pretrained models.

[xduzhangjiayu]

Hi, I did a guide for image fill, I commented at the end that you can use the same technique for out painting.

fffiloni which is very talentend and does some of the best spaces, took this knowledge and the space I made for it and implemented outpainting, you can check it here.

Even if this is not what I consider the final and the best I can do, a lot of people commented that this is the best outpainting they've seen and it's really fast. That space has a lot of cool additions too.

Also what do you mean by training-free? Even if the ones here are old and outdated, they were training-free, just inference with pretrained models.

Thanks for the reply!
Training-free is just use the base model (SDXL or SD3), not include ControlNet. Doesn't seem to have a this approach right now?

[asomoza]

The only way of doing what you're suggesting is with inpainting or differential diffusion, that approach always existed but it it's not good, as far as I know there is no good outpainting with just the base model.

Have you seen somewhere else a good outpainting with just the base model? if it exists, it would be easy to just port it to diffusers.

[xduzhangjiayu]

The only way of doing what you're suggesting is with inpainting or differential diffusion, that approach always existed but it it's not good, as far as I know there is no good outpainting with just the base model.

Have you seen somewhere else a good outpainting with just the base model? if it exists, it would be easy to just port it to diffusers.

Thanks!
I have tried sd3 inpainting using diffusers pipeline and sd3 base model, It works well for inpainting a small area, but if I Mask a large area, it doesn't perform as well.

[atharvasadanshive]

Code where the Error is detcted --->
zoe = ZoeDetector.from_pretrained("lllyasviel/Annotators")
image_zoe = zoe(white_bg_image, detect_resolution=512, image_resolution=1024)

Error --->>

TypeError: expected size to be one of int or Tuple[int] or Tuple[int, int] or Tuple[int, int, int], but got size with types [<class 'numpy.int64'>, <class 'numpy.int64'>]

Can someone please help me understand why am I am getting error in the zoe detector line of code.

[asomoza]

Hi, that's probably because of the image you're using but I'm more intrigued on why are you using this discussion to post a controlnet preprocessor issue you have, you probably won't get that much help here since this is an outdated discussion about outpainting.