Dalle Mini

🚀 DALL·E mini - Generate images from text

DALL·E mini is an attempt to replicate OpenAI's DALL·E, enabling the generation of arbitrary images from text prompts.

• Technical Report
• Demo

✨ Features

This model aims to replicate OpenAI's DALL·E, which can generate arbitrary images from text prompts. Its architecture simplifies the original, leveraging previous open - source efforts and pre - trained models. Although the results have lower quality than OpenAI's, it can be trained and used on less demanding hardware.

🔧 Technical Details

Components of the Architecture

The system is built on the Flax/JAX infrastructure, which is great for TPU training. However, TPUs are not necessary as both Flax and JAX run efficiently on GPU backends.

The main architecture components are as follows:

• Encoder: Based on BART. It transforms input text token sequences into image token sequences. Input tokens are extracted from the text prompt using the model's tokenizer. Image tokens are a fixed - length sequence representing indices in a VQGAN - based pre - trained codebook.
• Decoder: Converts image tokens to image pixels. It is based on a VQGAN model.

The encoder model definition can be downloaded from our Github repo. The encoder is represented by the class CustomFlaxBartForConditionalGeneration. To use the decoder, follow the instructions in our accompanying VQGAN model in the hub, flax-community/vqgan_f16_16384.

🚀 Quick Start

Basic Usage

The easiest way to get familiar with the code and models is by following the inference notebook in our github repo. You can open it in Google Colaboratory:

Advanced Usage

If you just want to test the trained model, visit our demo on 🤗 Spaces.

📚 Documentation

Our report provides more details about the model training process and showcases numerous examples demonstrating its capabilities.