Karlo V1 Alpha

🚀 Karlo v1 alpha

Karlo is a text-conditional image generation model. It's based on OpenAI's unCLIP architecture. It has an improved super - resolution model that can upscale from 64px to 256px and recover high - frequency details in just a few denoising steps.

• Original codebase

🚀 Quick Start

Karlo is available in diffusers! You can start using it by following the steps below.

📦 Installation

First, you need to install the necessary libraries:

pip install diffusers transformers accelerate safetensors

💻 Usage Examples

Basic Usage

Text to image

from diffusers import UnCLIPPipeline
import torch

pipe = UnCLIPPipeline.from_pretrained("kakaobrain/karlo-v1-alpha", torch_dtype=torch.float16)
pipe = pipe.to('cuda')

prompt = "a high-resolution photograph of a big red frog on a green leaf."

image = pipe([prompt]).images[0]

image.save("./frog.png")

Image variation

from diffusers import UnCLIPImageVariationPipeline
import torch
from PIL import Image

pipe = UnCLIPImageVariationPipeline.from_pretrained("kakaobrain/karlo-v1-alpha-image-variations", torch_dtype=torch.float16)
pipe = pipe.to('cuda')

image = Image.open("./frog.png")

image = pipe(image).images[0]

image.save("./frog-variation.png")

📚 Documentation

Model Architecture

Overview

Karlo is a text - conditional diffusion model based on unCLIP, which consists of prior, decoder, and super - resolution modules. In this repository, we have an improved version of the standard super - resolution module. It can upscale from 64px to 256px in only 7 reverse steps, as shown in the figure below:

Specifically, the standard SR module trained by the DDPM objective upscales 64px to 256px in the first 6 denoising steps using the respacing technique. Then, the additional fine - tuned SR module trained by VQ - GAN-style loss performs the final reverse step to recover high - frequency details. This approach is very effective for upscaling low - resolution images in a small number of reverse steps.

Details

We trained all components from scratch on 115M image - text pairs, including COYO - 100M, CC3M, and CC12M. For the Prior and Decoder, we used ViT - L/14 from OpenAI’s CLIP repository. To improve efficiency, we replaced the trainable transformer in the decoder with the text encoder in ViT - L/14, different from the original unCLIP implementation. For the SR module, we first trained the model using the DDPM objective for 1M steps, followed by an additional 234K steps to fine - tune the additional component. The following table summarizes the important statistics of our components:

In the checkpoint links, ViT - L - 14 is the same as the original version, but we included it for convenience. Note that ViT - L - 14 - stats is required to normalize the outputs of the prior module.

Evaluation

We quantitatively measured the performance of Karlo - v1.0.alpha on the validation split of CC3M and MS - COCO. The following tables show the CLIP - score and FID. To measure FID, we resized the shorter side of the image to 256px and then cropped it at the center. We set the classifier - free guidance scales for the prior and decoder to 4 and 8 in all cases. Our model achieved reasonable performance even with 25 sampling steps of the decoder.

CC3M

MS - COCO

For more information, please refer to the upcoming technical report.

Training Details

This alpha version of Karlo was trained on 115M image - text pairs, including the [COYO](https://github.com/kakaobrain/coyo - dataset)-100M high - quality subset, CC3M, and CC12M. If you're interested in a better version of Karlo trained on larger - scale high - quality datasets, please visit the landing page of our application B^DISCOVER.

📄 License

The model is released under the creativeml - openrail - m license.

BibTex

If you find this repository useful in your research, please cite:

@misc{kakaobrain2022karlo-v1-alpha,
  title         = {Karlo-v1.0.alpha on COYO-100M and CC15M},
  author        = {Donghoon Lee, Jiseob Kim, Jisu Choi, Jongmin Kim, Minwoo Byeon, Woonhyuk Baek and Saehoon Kim},
  year          = {2022},
  howpublished  = {\url{https://github.com/kakaobrain/karlo}},
}