How to set Fooocus parameters in Python?

[levoz92]

Hi.

I am interested in setting these parameters in Python:

    "default_loras": [[true,"SDXL_FILM_PHOTOGRAPHY_STYLE_BetaV0.4.safetensors",0.25],[true,"None",1.0],[true,"None",1.0],[true, "None",1.0],[true,"None",1.0]],
    "default_cfg_scale": 3.0,
    "default_sample_sharpness": 2.0,
    "default_sampler": "dpmpp_2m_sde_gpu",
    "default_scheduler": "karras",

Currently, this is how my code looks:

import inspect, torch, PIL, random, os
import numpy as np

from typing import List, Optional, Union
from diffusers import StableDiffusionInpaintPipeline, StableDiffusionXLInpaintPipeline, DDIMScheduler

device = "cuda"
model_path = "weights/realisticStockPhoto_v20.safetensors"
refiner_path = "weights/sd_xl_refiner_0.9.safetensors"
lora_path = "weights/SDXL_FILM_PHOTOGRAPHY_STYLE_BetaV0.4.safetensors"
img_url = "image.jpg"
mask_url = "mask.png"
prompt = 'on a desert highway, Detailed, High resolution, Photorealistic, Soft light, photograph, 50mm, cinematic 4k epic, shot on kodak, HBO dark moody, 35mm photo, grainy, vignette, vintage, Kodachrome, Lomography, stained, highly detailed, found footage'
guidance_scale=3
num_samples = 3
height = 896
width = 1152
high_noise_frac = 0.7
negative_prompt = 'unrealistic, saturated, high contrast, big nose, painting, drawing, sketch, cartoon, anime, manga, render, CG, 3d, watermark, signature, label, Brad Pitt, bokeh, depth of field, blurry, cropped, regular face, deformed iris, deformed pupils, semi-realistic, cgi, text, out of frame, worst quality, low quality, jpeg artifacts, ugly, duplicate, morbid, mutilated, extra fingers, mutated hands, poorly drawn hands, poorly drawn face, mutation, deformed, dehydrated, bad anatomy, bad proportions, extra limbs, cloned face, disfigured, gross proportions, malformed limbs, missing arms, missing legs, extra arms, extra legs, fused fingers, too many fingers, long neck, missing limb, floating limbs, disconnected limbs, disconnected head, malformed hands, mutated hands and fingers, bad hands, missing fingers, missing hand, disappearing arms, disappearing thigh, disappearing calf, disappearing legs, abnormal eye proportion, Abnormal hands, abnormal legs, abnormal feet, abnormal fingers, crayon, graphite, impressionist, noisy, soft, graphic, abstract, glitch'
num_inference_steps = 50
output_dir = "outputs"

pipe = StableDiffusionXLInpaintPipeline.from_single_file(model_path,torch_dtype=torch.float16,num_in_channels=4, use_safetensors=True).to(device)

pipe.load_lora_weights(".", weight_name=lora_path, adapter_name="film")
pipe.scheduler = DDIMScheduler.from_config(pipe.scheduler.config)

# refiner = StableDiffusionXLInpaintPipeline.from_single_file(refiner_path, text_encoder_2 = pipe.text_encoder_2, vae=pipe.vae, torch_dtype=torch.float16,num_in_channels=4, use_safetensors=True, variant="fp16").to(device)

generator = torch.Generator().manual_seed(random.randint(0,1000))

images = pipe(
    prompt=prompt,
    image=image,
    height = height,
    width = width,
    negative_prompt = negative_prompt,
    num_inference_steps = num_inference_steps,
    mask_image=mask_image,
    guidance_scale=guidance_scale,
    generator=generator,
    num_images_per_prompt=num_samples,
).images

# images = refiner(
#     prompt=prompt,
#     image=image,
#     mask_image=mask_image,
#     num_inference_steps=num_inference_steps,
#     denoising_start=high_noise_frac,
# ).images[0]

# Insert the initial image in the list so we can compare side by side
images.insert(0, image)

# Save images to the outputs directory
for i, img in enumerate(images):
    img.save(os.path.join(output_dir, f"image_{i}.png"))

# Print a message to confirm the images have been saved
print(f"All images have been saved to {output_dir}/")

I am trying to reach to the same inpainting quality of fooocus using Python. What more suggestions can you make?

[asomoza]

It depends on what you want, if you want the exact same results, there's an issue about this here:

#6529

Also fooocus does a lot more under the hood, is not enough to just copy the params, I haven't done any guides or tutorials with inpainting yet, but you can learn about some of them here:

prompt-enhancing-with-gpt2
Outpainting

If you have an example of what you are trying to achieve (original image and result) I can use it as material for my next guide if you want.

[asomoza]

well, the title of this discussion is "How to set Fooocus parameters in Python?" and was originally to get help on how to achieve the same fooocus results in diffusers. If this discussion goes beyond that, I'll rather like you open another discussion with the correct topic so people can find it (even you) in the future.

There are already 2 controlnets. Do you mean a 3rd one?

yes, you can add as many as you want, the line ones help with the small details and contours that the depth ones can miss, Mistoline is the best one in this regard.

[levoz92]

Adding a 3rd controlnet requires a 3rd image input. That's the error I got.

[asomoza]

yeah, if you add another controlnet you'll need to provide the corresponding image, in case of Mistoline you'll need to preprocess the source image with the AnyLine preprocessor.

[levoz92]

In this process, where and how does the AnyLine Preprocessor go?

 for i in range(n_of_images):
        seed = random.randint(0, 999999999)
        original_image_path = os.path.join(image_filename)
        original_image = Image.open(original_image_path).convert("RGBA")
        resized_img, white_bg_image = scale_and_paste(original_image)

        if not is_image(white_bg_image):
            raise TypeError(f"Expected image but got {type(white_bg_image)}")

        image_zoe = zoe(white_bg_image, detect_resolution=dim/2, image_resolution=dim)

        if not is_image(image_zoe):
            raise TypeError(f"Expected image but got {type(image_zoe)}")

        temp_image = generate_image(prompt_init, pipeline_1, negative_prompt_init, image_guidance_scale, img_generate_inference_steps, controlnet_conditioning_scale, control_guidance_end, white_bg_image, image_zoe, seed)

        x = (dim - resized_img.width) // 2
        y = (dim - resized_img.height) // 2
        temp_image.paste(resized_img, (x, y), resized_img)

        mask = Image.new("L", temp_image.size)
        mask.paste(resized_img.split()[3], (x, y))
        mask = ImageOps.invert(mask)
        final_mask = mask.point(lambda p: p > 128 and 255)

        mask_blurred = pipeline_2.mask_processor.blur(final_mask, blur_factor=0) # Mask Blur

        final_image = generate_outpaint(prompt, pipeline_2, negative_prompt, outpaint_guidance_scale, strength, outpaint_generate_inference_steps, temp_image, mask_blurred, seed)

[levoz92]

I wrote this function:

def generate_canny(image):
    image = load_image(image)
    image = np.array(image)
    image = cv2.Canny(image, 100, 200)
    image = image[:, :, None]
    image = np.concatenate([image, image, image], axis=2)
    image = Image.fromarray(image)
    return image # returns a canny image

but how do I use this in the way you're suggesting?