Pathstrimg format for DiffusersPipeline.from_pretrained()

[jdumke]

Hello, all.
I'm trying to load a local safetensors file into a pipeline to experiment with. But DiffussersPipline.from_pretained() complains about the pathstring and I can't figure out why.
Here is the script I use:

#!/usr/bin/env python3
# Let's get nuts with diffusers
import torch
#import torchvision
import time
from os import path as pathtools
from diffusers import DiffusionPipeline as DiPi
#from diffusers.utils import make_image_grid as Mig

pathToCheck = "/home/jds/test/fresch_ed/Easy-Diffusion-Linux/models/stable-diffusion/jd_tooned_v2_2023-10-02-0.71021.safetensors"
if torch.is_vulkan_available():
	dev = torch.device("vulkan:0")
else:
	dev = torch.device("cpu")

print(f"Using device {dev}")
if pathtools.exists(pathToCheck):
	modelPath = "file:/" + pathtools.abspath(pathToCheck)
	print(f"Requestes model {modelPath} exists and ready to load.")
	try:		
		pipe = DiPi.from_pretrained(modelPath).to(dev)
#		prompt = "Flower bouquet on festative table"
#		start = time.time()
#		images = pipe(prompt,	width=512, height=512, num_inference_steps=25, num_images_per_prompt=4,	generator=torch.manual_seed(0)).images
#		end = time.time()
#		print(f"Steted at: {start}\n Finished at: {end}\n Images generated in {end - start}")  
	except Exception as e:
		print(f"{e}")

and here is the shell output:

(env) $ python ./test_diffusers.py 
/home/jds/test/fresch_ed/Easy-Diffusion-Linux/installer_files/env/lib/python3.9/site-packages/transformers/utils/generic.py:311: FutureWarning: `torch.utils._pytree._register_pytree_node` is deprecated. Please use `torch.utils._pytree.register_pytree_node` instead.
  torch.utils._pytree._register_pytree_node(
/home/jds/test/fresch_ed/Easy-Diffusion-Linux/installer_files/env/lib/python3.9/site-packages/transformers/utils/generic.py:311: FutureWarning: `torch.utils._pytree._register_pytree_node` is deprecated. Please use `torch.utils._pytree.register_pytree_node` instead.
  torch.utils._pytree._register_pytree_node(
/home/jds/test/fresch_ed/Easy-Diffusion-Linux/installer_files/env/lib/python3.9/site-packages/diffusers/models/transformers/transformer_2d.py:34: FutureWarning: `Transformer2DModelOutput` is deprecated and will be removed in version 1.0.0. Importing `Transformer2DModelOutput` from `diffusers.models.transformer_2d` is deprecated and this will be removed in a future version. Please use `from diffusers.models.modeling_outputs import Transformer2DModelOutput`, instead.
  deprecate("Transformer2DModelOutput", "1.0.0", deprecation_message)
Using device vulkan:0
Requestes model file://home/jds/test/fresch_ed/Easy-Diffusion-Linux/models/stable-diffusion/jd_tooned_v2_2023-10-02-0.71021.safetensors exists and ready to load.
The provided pretrained_model_name_or_path "file://home/jds/test/fresch_ed/Easy-Diffusion-Linux/models/stable-diffusion/jd_tooned_v2_2023-10-02-0.71021.safetensors" is neither a valid local path nor a valid repo id. Please check the parameter.

Is there a special format for the pathstring or simmilar, which isn't explained anywhere? As it seen above I tried everything, especially URI-style local paths (file:// pseudo-protocol).

Any suggestions are welcome.
Greets,
jdumke

[alishanawer]

Hi Jdumke,

I think I might know the solution to your problem, DiffusionPipeline.from_pretrained() can’t load a single .safetensors file. It expects either a Hugging Face repo id (like "runwayml/stable-diffusion-v1-5") or a local folder in the diffusers format (with model_index.json, unet/, vae/, etc.). That’s why it says your path isn’t valid.

If you only have a .safetensors checkpoint, you need to convert it first. Diffusers provides a script called convert_original_stable_diffusion_to_diffusers.py in the repo’s scripts/ folder. The typical usage looks like this:

python scripts/convert_original_stable_diffusion_to_diffusers.py \
  --checkpoint_path /home/jds/.../jd_tooned_v2_2023-10-02-0.71021.safetensors \
  --dump_path /home/jds/models/jd_tooned_v2-diffusers \
  --original_config_file /path/to/v1-inference.yaml

After that, you point from_pretrained() to the new folder:

pipe = DiffusionPipeline.from_pretrained(
    "/home/jds/models/jd_tooned_v2-diffusers"
).to(dev)

Alternatively, if you’re on a recent version of diffusers, you can use:

from diffusers import DiffusionPipeline
pipe = DiffusionPipeline.from_single_file(
    "/home/jds/.../jd_tooned_v2_2023-10-02-0.71021.safetensors"
).to(dev)

So in short: either convert the checkpoint into a folder and load it, or use from_single_file() if your version supports it.

Hope this helps ✌️

[jdumke]

Thanks a lot. No tutorial I read explained this differences. Where can I find the mentioned yaml-file?

[asomoza]

actually it's a lot easier, you can just use from_single_file to load it if you use the corresponding pipeline instead of DiffusionPipeline (since it seems to be a SD 1.5 model it should be StableDiffusionPipeline), later if you want it as a diffusers format model, you can then save it with save_pretrained.

[asomoza]

if you still want to do it the hard way, you can find the yaml file in the unofficial repo of the model here

[jdumke]

Thanks for the tips. Hence my GPU isn't supported by ROCm I've to do the hardway to do some work with my models.

[asomoza]

I don't quite get it, you can use from_single_file and save_pretrained with just the CPU, at this point, the GPU doesn't matter, but if it works for you using the other way, it's also good.