Knowledgebase

Introduction

Stable Diffusion Turbo models, Stable Diffusion (SD) Turbo and Stable Diffusion XL (SDXL) Turbo are real-time revolutionary text-to-image generation models developed by Stability AI. They're are based on the previous Stable Diffusion 2.1 and Stable Diffusion XL (SDXL) models with a new execution Adversarial Diffusion Distillation (ADD) technique.

ADD allows these models to synthesize image outputs in a series of one to four steps as compared to the conventional 50 steps. This remarkable capability makes both models much faster than previous text-to-image models and image-to-image models, which typically require hundreds of steps to generate an image.

This guide explains how you can achieve AI generated images using Stable Diffusion Turbo Models on a Rcs Cloud GPU server. You are to install the required packages to create the development environment and further compare the model time differences, performance benefits, and limitations while generating images.

Prerequisites

Before you begin, you should:

• Deploy a fresh Ubuntu 22.04 A100 Rcs GPU Stack server with at least 80 GB GPU RAM
• Securely access the server using SSH as a non-root sudo user
• Update the server
• Access the JupyterLab Web Interface

Adversarial Diffusion Distillation (ADD)

Adversarial Diffusion Distillation (ADD) is an advanced training method for large-scale image diffusion models such as Stable Diffussion Turbo. It utilizes score distillation with a pre-trained Stable Diffusion model as a guide to distill valuable knowledge for a high-quality output. The inclusion of an adversarial loss mechanism ensures realism which makes ADD a potent tool for real-time image generation.

ADD speeds up SD Turbo models by reducing the refinement steps through knowledge distillation from a pre-trained "teacher" model. This dual approach leverages the teacher's knowledge, results in real-time image generation, lower computational costs, and quicker experimentation even in the absence of official code.

Install the Model Dependency Packages

1. Open a terminal within the Jupyter lab interface.

   

2. Install the required dependency packages.

   console Copy

   $ pip install diffusers transformers accelerate matplotlib ipywidgets --upgrade
   

   The above command installs the following packages:

   • diffusers: Provides tools for building and training diffusion models. Diffusers include many different core neural network models and are used as building blocks to create new pipelines.
   • transformers: Consists of multiple pre-trained models used for Natural Language Processing (NLP), Named Entity Recognition (NER), machine translation, and sentiment analysis.
   • accelerate: Enables PyTorch to run across any distributed configuration. It uses accelerators such as GPUs and TPUs to improve efficiency and scalability, speed up natural language processing (NLP) workflows, and enhance performance.
   • matplotlib: Display the images in your Jupyter Notebook.

Generate Images using Stable Diffusion Turbo

Stable Diffusion Turbo is available as a pre-trained model with the Hugging Face Diffusers library. The model can generate images from text prompts and images from images. In this section, generate images using the model using both prompt types as described in the steps below.

Text to Image Generation

To generate images from text prompts, import the required modules such as AutoPipelineForText2Image from diffusers, torch, and matplotlib.pyplot. Then, initialize the model using the provided model_id to set it up for GPU acceleration by assigning it to the CUDA device.

1. Open a new Notebook session, and set its name to Stable Diffusion Turbo Text2Image.

   

2. To use the model, import the following packages.

   python Copy

   import torch
   import matplotlib.pyplot as plt
   from diffusers import AutoPipelineForText2Image
   

   Below is what each module does:

   • AutoPipelineForText2Image: Provides an interface to the model for generating images.
   • torch: Enables support for tensor computations. In this context, it's used for GPU acceleration.
   • matplotlib: Allows you to display the generated images.

3. Declare the model.

   python Copy

   model_id = "stabilityai/sd-turbo"
   pipe = AutoPipelineForText2Image.from_pretrained(
       model_id, 
       torch_dtype=torch.float16, 
       variant="fp16"
   )
   pipe.to("cuda")
   

   The pipeline applies the necessary setup to generate images from text using the from_pretrained method. The parameters passed to the method include:

   • model_id: Loads the stabilityai/sd-turbo model. The model ID can also be the path to a local directory containing model weights or URL to a checkpoint file.
   • torch_dtype: Defines the Torch datatype of the tensors to use for pipeline computations. float16 is specified explicitly to run model computations with 16-bit floating point numbers. To let the system choose the optimal data type use the auto value torch_dtype = "auto".

4. Generate an image using a text prompt. For example, Sunset on a beach.

   python Copy

   prompt = "Sunset on a beach"
   image = pipe(prompt=prompt, num_inference_steps=1, guidance_scale=0.0).images
   

   The above code declares and feeds the prompt to the previously declared pipeline to store the image attribute. A different image is generated each time you run the module. To enhance the image output, enter a more detailed prompt to match your desired results.

5. View the generated image.

   python Copy

   plt.imshow(image[0])
   

   Verify that the generated image matches your Sunset on a Beach prompt or fine-tune the prompt to generate a more desired result.

   

Image to Image Generation

1. Open a new Notebook and set its name to Stable Diffusion Turbo Image2Image.

   

2. Navigate to the Kernel menu option in your Jupyter Notebook, and click Shutdown Down All Kernels to clear GPU memory.

   

3. In a new code cell, import the following packages.

   python Copy

   import torch
   import matplotlib.pyplot as plt
   from diffusers import AutoPipelineForImage2Image
   from diffusers.utils import load_image
   

   Below is what each module does:

   • AutoPipelineForImage2Image: Provides an interface to the model for refining images.
   • load_image: Loads image data from a declared source.

4. Declare the model.

   python Copy

   model_id = "stabilityai/sd-turbo"
   pipe = AutoPipelineForImage2Image.from_pretrained(
       model_id, 
       torch_dtype=torch.float16, 
       variant="fp16"
   )
   pipe.to("cuda")
   

5. Load the image to the pipeline. Replace https://i.imgur.com/wFC9Yw6.png with your desired source image URL.

   python Copy

   init_image = load_image("https://i.imgur.com/wFC9Yw6.png").resize((512, 512))
   

6. Add a supporting text prompt to guide the image generation process. For example, A fantasy landscape, trending on artstation.

   python Copy

   prompt = "A fantasy landscape, trending on artstation"
   image = pipe(prompt, image=init_image, num_inference_steps=4, strength=1, guidance_scale=0.0).images
   

7. View the generated image.

   python Copy

   plt.imshow(image[0])
   

   Verify that your input image matched with the supporting prompt generates a new image.

   

Time Difference Compared to Stable Diffusion 2.1

Text to Image Generation Time Difference

In this section, compare the text-to-image generation times between SD Turbo and Stable Diffusion 2.1. The image generation steps are similar, but you need to import the time variable to calculate the image generation duration for each model as described in the steps below.

1. Open a new Notebook and set its name to Stable Diffusion text-to-image Time Comparision.

   

2. Clear the GPU memory to start running the model

   

3. Import the following packages.

   python Copy

   import torch
   import matplotlib.pyplot as plt
   from diffusers import AutoPipelineForText2Image
   

4. Declare the model.

   python Copy

   model_id = "stabilityai/sd-turbo"
   pipe = AutoPipelineForText2Image.from_pretrained(
       model_id, 
       torch_dtype=torch.float16, 
       variant="fp16"
   )
   pipe.to("cuda")
   

5. Generate an image using the Stable Diffusion Turbo model with a prompt such as Sunset on a beach.

   python Copy

   import time
   prompt = "Sunset on a beach"
   start_time = time.time()
   
   image = pipe(prompt=prompt, num_inference_steps=1, guidance_scale=0.0).images
   end_time = time.time()
   plt.imshow(image[0])
   

6. View the model generation time.

   python Copy

   elapsed_time = end_time - start_time
   print(f"Image generation time: {elapsed_time} seconds")
   

   Output:

   Image generation time: 0.7282497882843018 seconds

   Keep note of the time value.

7. Generate an image with the Stable Diffusion 2.1 Model.

   The Stable Diffusion 2.1 model steps are similar to SD Turbo text-to-image generation. In the previous code examples, replace the model_id name and remove num_inference_steps, and guidance_scale variables to pass the prompt with its default setting as described below.

   Replace the model_ID.

   python Copy

   model_id = "stabilityai/stable-diffusion-2-1"
   

   Generate an image by providing using a prompt such as Sunset on a beach.

   python Copy

   import time
   prompt = "Sunset on a beach"
   start_time = time.time()
   
   image = pipe(prompt=prompt).images
   end_time = time.time()
   plt.imshow(image[0])
   

8. Print the time.

   python Copy

   elapsed_time = end_time - start_time
   print(f"Image generation time: {elapsed_time} seconds")
   

   Output:

   Image generation time: 6.339369297027588 seconds

In comparison with the earlier Stable Diffusion Turbo 0.7282497882843018 seconds time, Stable Diffusion 2.1 is slower with 6.339369297027588 seconds. This is because SD Turbo can generate images in just one step, while Stable Diffusion 2.1 requires 50 steps to generate an image.

Image to Image Generation Time Difference

1. Open a new Notebook and set its name to Stable Diffusion image-to-image Time Comparision.

   

2. Clear the GPU memory to start running the model.

   

3. Import the following packages.

   python Copy

   import torch
   import matplotlib.pyplot as plt
   from diffusers import AutoPipelineForImage2Image
   from diffusers.utils import load_image
   

4. Declare the model.

   python Copy

   model_id = "stabilityai/sd-turbo"
   pipe = AutoPipelineForImage2Image.from_pretrained(
       model_id, 
       torch_dtype=torch.float16, 
       variant="fp16"
   )
   pipe.to("cuda")
   

5. Generate an image with the SD Turbo Model.

   python Copy

   import time
   prompt = "A fantasy landscape, trending on artstation"
   start_time = time.time()
   
   image = pipe(prompt, image=init_image, num_inference_steps=4, strength=1, guidance_scale=0.0).images
   end_time = time.time()
   plt.imshow(image[0])
   

6. Print the time.

   python Copy

   elapsed_time = end_time - start_time
   print(f"Image generation time: {elapsed_time} seconds")
   

   Output:

   Image generation time: 0.8927862644195557 seconds

   Keep note of the generation time.

7. Generate an image with the Stable Diffusion 2.1 Model.

   Replace the model_ID.

   python Copy

   model_id = "stabilityai/stable-diffusion-2-1"
   

   Generate an image with a prompt such as A fantasy landscape, trending on artstation.

   python Copy

   import time
   prompt = "A fantasy landscape, trending on artstation"
   start_time = time.time()
   
   image = pipe(prompt, image=init_image).images
   end_time = time.time()
   plt.imshow(image[0])
   

8. Print the time.

   python Copy

   elapsed_time = end_time - start_time
   print(f"Image generation time: {elapsed_time} seconds")
   

   Output:

   Image generation time: 2.755378007888794 seconds

Based on the image generation time, SD Turbo is much faster than Stable Diffusion 2.1. SD Turbo creates an in four steps, while Stable Diffusion 2.1 uses 40 steps for an image.

Generate Images using Stable Diffusion XL Turbo

SDXL Turbo is similar to the SD Turbo model and is a larger version capable of generating higher quality and clearer images. Similarly, it can also be used to generate images from images.

Text to Image Generation

The SDXL Turbo model steps are similar to the SD Turbo text-to-image generation process. In the previous code examples, replace the model_id name as below.

1. Open a new Notebook and set its name to Stable Diffusion XL Turbo Text2Image.

   

2. To clear GPU memory and start running the model, navigate to the Kernel menu option in your Jupyter Notebook, and click Shutdown Down All Kernels.

   

3. Import the following packages.

   python Copy

   import torch
   import matplotlib.pyplot as plt
   from diffusers import AutoPipelineForText2Image
   

4. Declare the model.

   python Copy

   model_id = "stabilityai/sdxl-turbo"
   pipe = AutoPipelineForText2Image.from_pretrained(
       model_id, 
       torch_dtype=torch.float16, 
       variant="fp16"
   )
   pipe.to("cuda")
   

5. Generate an image by providing a prompt as below.

   python Copy

   prompt = "A baby racoon wearing a robe"
   image = pipe(prompt=prompt, num_inference_steps=1, guidance_scale=0.0).images
   

   Replace A baby raccoon wearing a robe with your desired prompt.

6. Render the generated image.

   python Copy

   plt.imshow(image[0])
   

   The A baby raccoon wearing a robe generates an image like the one below:

   

7. You can also access the above code in the following Jupyter Notebook.

   Stable Diffusion XL Turbo Text2Image Notebook

Image to Image Generation

Stable Diffusion XL Turbo model steps are similar to the SD Turbo image-to-image generation process. In the previous code examples, replace the model_id variable with the SDXL Turbo model to generate images as described in the steps below.

1. Open a new Notebook and set its name to Stable Diffusion XL Turbo Image2Image.

   

2. Navigate to the Kernel menu option in your Jupyter Notebook, and click Shutdown Down All Kernels to clear GPU memory.

   

3. Import the following packages.

   python Copy

   import torch
   import matplotlib.pyplot as plt
   from diffusers import AutoPipelineForImage2Image
   from diffusers.utils import load_image
   

4. Declare the model.

   python Copy

   model_id = "stabilityai/sdxl-turbo"
   pipe = AutoPipelineForImage2Image.from_pretrained(
       model_id, 
       torch_dtype=torch.float16, 
       variant="fp16"
   )
   pipe.to("cuda")
   

5. Load the image.

   python Copy

   init_image = load_image("https://i.imgur.com/5OiqkNA.png").resize((512, 512))
   

6. Generate an image using a text prompt such as Astronauts in a jungle.

   python Copy

   prompt = "Astronauts in a jungle"
   image = pipe(prompt, image=init_image, num_inference_steps=4, strength=1, guidance_scale=0.0).images
   

7. View the generated image.

   python Copy

   plt.imshow(image[0])
   

   Verify that your generated image matches your input prompt Astronauts in a Jungle.

   

SDXL Turbo and XDXL Text to Image Generation Time Differences

1. Open a new Notebook and set its name to Stable Diffusion XL text-to-image Time Comparision.

   

2. Clear the GPU memory to start running the model

   

3. Import the following packages.

   python Copy

   import torch
   import matplotlib.pyplot as plt
   from diffusers import AutoPipelineForText2Image
   

4. Declare the model.

   python Copy

   model_id = "stabilityai/sdxl-turbo"
   pipe = AutoPipelineForText2Image.from_pretrained(
       model_id, 
       torch_dtype=torch.float16, 
       variant="fp16"
   )
   pipe.to("cuda")
   

5. Generate an image with the SDXL Turbo Model.

   python Copy

   import time
   prompt = "A baby racoon wearing a robe."
   start_time = time.time()
   
   image = pipe(prompt=prompt, num_inference_steps=1, guidance_scale=0.0).images
   end_time = time.time()
   plt.imshow(image[0])
   

6. Print the time.

   python Copy

   elapsed_time = end_time - start_time
   print(f"Image generation time: {elapsed_time} seconds")
   

   Output:

   Image generation time: 0.8553969860076904 seconds

7. Generate an image with the SDXL Model.

   Replace the model_id.

   python Copy

   model_id = "stabilityai/stable-diffusion-xl-base-1.0"
   

   Generate an image using a prompt such as A baby racoon wearing a robe.

   python Copy

   import time
   prompt = "A baby racoon wearing a robe"
   start_time = time.time()
   
   image = pipe(prompt=prompt).images
   end_time = time.time()
   plt.imshow(image[0])
   

8. Print the time.

   python Copy

   elapsed_time = end_time - start_time
   print(f"Image generation time: {elapsed_time} seconds")
   

   Output:

   Image generation time: 13.608830213546753 seconds

Image to Image Generation Time Differences

1. Open a new Notebook and set the filename to Stable Diffusion XL image-to-image Time Comparision.

   

2. Clear the GPU memory to start running the model

   

3. Import the following packages.

   python Copy

   import torch
   import matplotlib.pyplot as plt
   from diffusers import AutoPipelineForImage2Image
   from diffusers.utils import load_image
   

4. Declare the model.

   python Copy

   model_id = "stabilityai/sdxl-turbo"
   pipe = AutoPipelineForImage2Image.from_pretrained(
       model_id, 
       torch_dtype=torch.float16, 
       variant="fp16"
   )
   pipe.to("cuda")
   

5. Generate an image with the SDXL Turbo Model.

   python Copy

   import time
   prompt = "Astronauts in a jungle"
   start_time = time.time()
   
   image = pipe(prompt, image=init_image, num_inference_steps=4, strength=1, guidance_scale=0.0).images
   end_time = time.time()
   plt.imshow(image[0])
   

6. Print the time.

   python Copy

   elapsed_time = end_time - start_time
   print(f"Image generation time: {elapsed_time} seconds")
   

   Output:

   Image generation time: 1.2138450145721436 seconds

7. Generate an image with the SDXL Model.

   Replace the model_id.

   python Copy

   model_id = "stabilityai/stable-diffusion-xl-base-1.0"
   

   Generate an image using a prompt such as Astronauts in a jungle

   python Copy

   import time
   prompt = "Astronauts in a jungle"
   start_time = time.time()
   
   image = pipe(prompt, image=init_image, strength=0.8, guidance_scale=10.5).images
   end_time = time.time()
   plt.imshow(image[0])
   

8. Print the time.

   python Copy

   elapsed_time = end_time - start_time
   print(f"Image generation time: 
   
                       
   
               
   Was this answer helpful?
               
   
   
                   
                   
   
                       
                           
                           Yes
                       
                       
                           
                           No
                       
                   
               
           
       

   Related Articles

   How to Implement Apache Cassandra Driver for Python Introduction Apache Cassandra is a fault-tolerant high-availability NoSQL database. The... How to Cache MySQL Data with Redis and Python on Ubuntu 20.04 Introduction You can improve the reliability of Python applications by caching frequently... How to Use MongoDB in Python with MongoEngine .article--sm, .article--sm .alert { font-size: 15px; line-height: 24px;... How to Cache MySQL Data with Redis and Python on Ubuntu 20.04 .article--sm, .article--sm .alert { font-size: 15px; line-height: 24px;... How to Securely Connect to Redis with TLS/SSL in Go, NodeJS, PHP, Python, and redis-cli .article--sm, .article--sm .alert { font-size: 15px; line-height: 24px;...

   Back

   Powered by WHMCompleteSolution