TIPO 500M

🚀 TIPO: Text to Image with text presampling for Prompt Optimization

An innovative framework enhancing text-to-image generative models by refining user prompts with LLMs.

🚀 Quick Start

Use the updated version of the DTG extension (renamed to z - tipo - extension). The current version of z - tipo - extension supports stable - diffusion - webui, stable - diffusion - webui - forge, and ComfyUI. SD - Next hasn't been tested. z - tipo - extension on GitHub

✨ Features

In this project, we introduce "TIPO" (Text to Image with text presampling for Prompt Optimization), an innovative framework designed to significantly enhance the quality and usability of Text - to - Image (T2I) generative models. TIPO utilizes the Large Language Models (LLMs) to perform "Text Presampling" within the inference pipeline of text - to - image generative modeling. By refining and extending user input prompts, TIPO enables generative models to produce superior results with minimal user effort, making T2I systems more accessible and effective for a wider range of users.

🔧 Technical Details

Model arch and Training

This model is LLaMA arch with 200M parameters, and the training data is a combined version of Danbooru2023 and Coyo - HD - 11M. The total token seen is around 50B tokens. For more information, please refer to the tech report and the following table.

*: We only count "non - padding token" in the token seen, since all the training data have a very large length range. `: Since the training data is pretty short, it takes more time to reach the same token seen than general LLM pretraining. As a reference, with 4096 as the max ctx length and almost all the data reaching that length, you may only need 2 days to reach 10B token seen on RTX 3090 x 4 with a 200M model.

Evaluation

Evaluation is done on the TIPO - 200M model We have tested TIPO compared to other models in several tests and metrics:

Scenery tag test

In this test, we use a single "scenery" tag as input (with some certain meta). To test each prompt gen method to see if they can obtain the desired distribution of outputs while maintaining the quality of images.

Short/Truncated Long test

In this test, we use short captions or manually truncated captions from GBC10M and CoyoHD11M. This test examines the ability of the prompt gen method to handle almost completed prompts.

📄 License

This model is released under Kohaku License 1.0. You can check the above - provided URL or check the LICENSE file in this repo.

Citation

@misc{yeh2024tipotextimagetext,
      title={TIPO: Text to Image with Text Presampling for Prompt Optimization}, 
      author={Shih - Ying Yeh and Sang - Hyun Park and Giyeong Oh and Min Song and Youngjae Yu},
      year={2024},
      eprint={2411.08127},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2411.08127}, 
}