Stable Diffusion 1.5

🚀 Stable Diffusion v1-5 Model Card

Stable Diffusion is a latent text-to-image diffusion model that can generate photo-realistic images from any text input. For more details on how it works, refer to 🤗's Stable Diffusion blog.

🚀 Quick Start

The Stable-Diffusion-v1-5 checkpoint was initialized with the weights of Stable-Diffusion-v1-2 and fine-tuned on 595k steps at 512x512 resolution on "laion-aesthetics v2 5+". 10% of text-conditioning was dropped to enhance classifier-free guidance sampling.

You can use this model with both the 🧨Diffusers library and the RunwayML GitHub repository.

✨ Features

• Capable of generating photo-realistic images from text input.
• Can be used with different libraries and repositories for various applications.

📦 Installation

N/A

💻 Usage Examples

Basic Usage

from diffusers import StableDiffusionPipeline
import torch

model_id = "runwayml/stable-diffusion-v1-5"
pipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16)
pipe = pipe.to("cuda")

prompt = "a photo of an astronaut riding a horse on mars"
image = pipe(prompt).images[0]  
    
image.save("astronaut_rides_horse.png")

For more detailed instructions, use-cases, and JAX examples, follow the guide here.

Advanced Usage

N/A

📚 Documentation

Model Details

Uses

Direct Use

The model is for research purposes only. Potential research areas and tasks include:

• Safe deployment of models with the potential to generate harmful content.
• Probing and understanding the limitations and biases of generative models.
• Artwork generation and use in design and other artistic processes.
• Applications in educational or creative tools.
• Research on generative models.

Excluded uses are detailed below.

Misuse, Malicious Use, and Out-of-Scope Use

Note: This section is from the DALLE-MINI model card and applies to Stable Diffusion v1 as well.

The model should not be used to intentionally create or spread images that create hostile or alienating environments. This includes generating disturbing, distressing, or offensive images, or content that propagates stereotypes.

Out-of-Scope Use

The model was not trained to represent people or events factually. Using it to generate such content is beyond its capabilities.

Misuse and Malicious Use

Using the model to generate cruel content towards individuals is a misuse. This includes, but is not limited to:

• Generating demeaning, dehumanizing, or harmful representations of people, their environments, cultures, religions, etc.
• Intentionally promoting or propagating discriminatory content or harmful stereotypes.
• Impersonating individuals without consent.
• Generating sexual content without the consent of those who may see it.
• Spreading mis- and disinformation.
• Representing egregious violence and gore.
• Sharing copyrighted or licensed material in violation of its terms of use.
• Sharing altered copyrighted or licensed material in violation of its terms of use.

Limitations and Bias

Limitations

• It doesn't achieve perfect photorealism.
• It can't render legible text.
• It performs poorly on complex tasks involving compositionality, like rendering "A red cube on top of a blue sphere".
• Faces and people may not be generated correctly.
• It was mainly trained with English captions and works less well in other languages.
• The autoencoding part is lossy.
• It was trained on LAION-5B, which contains adult material and isn't suitable for product use without additional safety measures.
• No deduplication was done on the dataset, leading to some memorization of duplicated training images. Search the training data at https://rom1504.github.io/clip-retrieval/ to detect memorized images.

Bias

Image generation models can reinforce or exacerbate social biases. Stable Diffusion v1 was trained on subsets of LAION-2B(en), mainly with English descriptions. Texts and images from non-English communities are underrepresented, affecting the output. White and western cultures are often the default, and the model performs worse with non-English prompts.

Safety Module

The model is intended to be used with the Safety Checker in Diffusers. This checker compares model outputs against known hard-coded NSFW concepts. The concepts are hidden to prevent reverse-engineering. Specifically, it compares the class probability of harmful concepts in the embedding space of the CLIPTextModel after image generation, using hand-engineered weights for each NSFW concept.

Training

Training Data

The model was trained on:

• LAION-2B (en) and its subsets (see next section)

Training Procedure

Stable Diffusion v1-5 is a latent diffusion model combining an autoencoder and a diffusion model trained in the autoencoder's latent space. During training:

• Images are encoded by an encoder into latent representations. The autoencoder downsamples by a factor of 8, mapping H x W x 3 images to H/f x W/f x 4 latents.
• Text prompts are encoded by a ViT-L/14 text-encoder.
• The non-pooled output of the text encoder is fed into the UNet backbone of the latent diffusion model via cross-attention.
• The loss is a reconstruction objective between the added noise in the latent and the UNet's prediction.

Six Stable Diffusion checkpoints were trained as follows:

• stable-diffusion-v1-1: 237,000 steps at 256x256 on laion2B-en, then 194,000 steps at 512x512 on laion-high-resolution (170M examples from LAION-5B with resolution >= 1024x1024).

• stable-diffusion-v1-2: Resumed from stable-diffusion-v1-1, 515,000 steps at 512x512 on "laion-improved-aesthetics" (a filtered subset of laion2B-en).

• stable-diffusion-v1-3: Resumed from stable-diffusion-v1-2, 195,000 steps at 512x512 on "laion-improved-aesthetics" with 10% text-conditioning dropping for classifier-free guidance sampling.

• stable-diffusion-v1-4: Resumed from stable-diffusion-v1-2, 225,000 steps at 512x512 on "laion-aesthetics v2 5+" with 10% text-conditioning dropping for classifier-free guidance sampling.

• stable-diffusion-v1-5: Resumed from stable-diffusion-v1-2, 595,000 steps at 512x512 on "laion-aesthetics v2 5+" with 10% text-conditioning dropping for classifier-free guidance sampling.

• stable-diffusion-inpainting: Resumed from stable-diffusion-v1-5, then 440,000 steps of inpainting training at 512x512 on “laion-aesthetics v2 5+” with 10% text-conditioning dropping. For inpainting, the UNet has 5 additional input channels, and their weights were zero-initialized after restoring the non-inpainting checkpoint. During training, synthetic masks were generated, and 25% of the image was masked.

• Hardware: 32 x 8 x A100 GPUs

• Optimizer: AdamW

• Gradient Accumulations: 2

• Batch: 32 x 8 x 2 x 4 = 2048

• Learning rate: Warmed up to 0.0001 for 10,000 steps and then kept constant

Evaluation Results

Evaluations with different classifier-free guidance scales (1.5, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0) and 50 PNDM/PLMS sampling steps show the relative improvements of the checkpoints:

📄 License

This model is open access under a CreativeML OpenRAIL-M license. The license specifies:

1. You can't use the model to deliberately produce or share illegal or harmful outputs or content.
2. CompVis claims no rights on the outputs you generate. You are free to use them but accountable for their use, which must comply with the license.
3. You may re-distribute the weights and use the model commercially or as a service. If you do, include the same use restrictions and share a copy of the CreativeML OpenRAIL-M with all users.

Please read the full license here.