diffusion/diffusers/stable_diffusion_with_diffusers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stable Diffusion with Diffusers\n",
    "\n",
    "Stable Diffusion is a text-to-image latene diffusion model created by the researchers and engineers from CompVis, Stability AI and LAION. It is trained on 512x512 images from a subset of the LAION-5B database. This model uses a frozen CLIP ViT-L/14 text encoder to condition the model on text prompts. With its 860M Unet and 123M text encoder, the model is realatively lightweight and can run on many consumer GPUs.\n",
    "\n",
    "LAION-5B is the largest, freely accessible multi-model dataset that currently exists.\n",
    "\n",
    "Here we use Stable Diffusion witht the 🤗 diffusers library, explain how the model works and finally dive a bit deeper into how diffusers allows once to customize the image generation pipeline. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How to use StableDiffusionPipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, please make sure you are using a GPU runtime to run this notebook, so inference is much faster. If the following commend fails, use the `Runtime` menu above and select `Change runtime type`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os, platform\n",
    "\n",
    "torch_device = 'cpu'\n",
    "\n",
    "if 'kaggle' in os.environ.get('KAGGLE_URL_BASE', 'localhost'):\n",
    "    torch_device = 'cuda'\n",
    "else:\n",
    "    torch_device = 'mps' if platform.system() == 'Darwin' else 'cpu'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch_device"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.environ['PYTORCH_ENABLE_MPS_FALLBACK'] = '1'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, you should install diffusers as well scipy, ftfy and transformers.accelerate is used to achieve much faster loading."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install --upgrade git+https://github.com/huggingface/diffusers.git"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install transformers scipy ftfy accelerate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stable Diffusion Pipeline\n",
    "\n",
    "`StableDiffusionPipeline` is an end-to-end inference pipeline that you can yse to generate images from text with just a few lines of code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from diffusers import StableDiffusionPipeline, DPMSolverMultistepScheduler\n",
    "\n",
    "repo_id = 'stabilityai/stable-diffusion-2-1'\n",
    "pipe = StableDiffusionPipeline.from_pretrained(repo_id, torch_dtype=torch.float16)\n",
    "pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe = pipe.to(torch_device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt ='house, shot 35 mm, realism, octane render, 8k, trending on artstation, 35 mm camera, unreal engine, hyper detailed, photo - realistic maximum detail, volumetric light, realistic matte painting, hyper photorealistic, trending on artstation, ultra - detailed, realistic'\n",
    "negative_prompt='BadDream, (UnrealisticDream:1.3)'\n",
    "\n",
    "image = pipe(prompt=prompt, negative_prompt=negative_prompt).images[0] # image here is in [PIL format](https://pillow.readthedocs.io/en/satble/)\n",
    "\n",
    "# Now to display an image you can either save it such as:\n",
    "image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Seed](https://aisuko.gitbook.io/wiki/ai-techniques/stable-diffusion/the-important-parameters-for-stunning-ai-image#seed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Running the above cell multiple times will give you a different image every time. If you want deterministic output you can pass a random seed to the pipeline. Every time you use the same you will have the same image result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "generator = torch.Generator(torch_device).manual_seed(5775709)\n",
    "\n",
    "image = pipe(prompt=prompt,negative_prompt=negative_prompt, generator=generator).images[0]\n",
    "\n",
    "image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### num_inference_steps\n",
    "\n",
    "We can change the number of inference steps using the `num_inference_steps` argument. In general, results are better the more steps you use. Normally yhe default value is 50. If you want faster results you can use a smaller number."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "generator = torch.Generator(torch_device).manual_seed(5775709)\n",
    "\n",
    "image = pipe(prompt=prompt, negative_prompt=negative_prompt, num_inference_steps=30, generator=generator).images[0]\n",
    "\n",
    "image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generating multiple images\n",
    "\n",
    "It is a way to increase the adherence to the conditional signal which in this case is text as well as overall sample quality. In simple terms classifer free guidance dorces the generation to better match with the prompt. Numbers like 7 or 8.5 give good results, if you use a very large number the images might look good, but will be less diverse."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from PIL import Image\n",
    "\n",
    "def image_grid(imgs, rows, cols):\n",
    "    assert len(imgs) == rows*cols\n",
    "\n",
    "    w, h = imgs[0].size\n",
    "    grid = Image.new('RGB', size=(cols*w, rows*h))\n",
    "    grid_w, grid_h = grid.size\n",
    "    \n",
    "    for i, img in enumerate(imgs):\n",
    "        grid.paste(img, box=(i%cols*w, i//cols*h))\n",
    "    return grid"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_images = 3\n",
    "prompt1 = [\"a photograph of an astronaut riding a horse\"] * num_images\n",
    "\n",
    "images = pipe(prompt1).images\n",
    "\n",
    "grid = image_grid(images, rows=1, cols=3)\n",
    "grid"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate non_square images\n",
    "\n",
    "Let's create a rectangular images in portrait or landscape ratios, there are some recommendations to chooise good image sizes:\n",
    "* Make sure `heigh` and `width` are both multiples of 8\n",
    "* Going below 512 might result in lower quality images\n",
    "* Going over 512 in both directions will repeat image areas (global coherence is lost)\n",
    "* The best way to create non-squre images is to use `512` in one dimension, and a value larger than that in the other one"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "image = pipe(prompt=prompt, num_inference_steps=30, generator=generator, height=512, width=768).images[0]\n",
    "image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Stable Diffusion during inference\n",
    "\n",
    "Putting it together, let's now takes a closer look at how the model works in inference by illustrating the logical flow\n",
    "\n",
    "<p align=\"center\">\n",
    "    <img src=\"https://hostux.social/system/media_attachments/files/110/683/631/285/614/442/original/6a9f7fecd5e3949b.png\" width=\"1000\" />\n",
    "</p>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The stable diffusion model takes both a latent seed and a text prompt as an input. The latent seed is then used to generate random latent image representations of size 64x64 where as the text prompt is tranformed to text embeddings of size 77x768 via CLIP's text encoder.\n",
    "\n",
    "DPM Solve Multistep scheduler is able to achieve great quality in less steps, like 25."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "diffusers",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.3"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}