Bloom-560m-RLHF-SD2-prompter Open Source Model - Automatically Expand and Generate High-Quality Image Descriptions

Bloom 560m RLHF SD2 Prompter

Developed by crumb

An RLHF fine-tuned Stable Diffusion 2.0 prompt generation model that can automatically expand or generate high-quality image descriptions

Text Generation

Transformers

Open Source License:Openrail #SD2 Prompt Optimization #RLHF Fine-tuning #Creative Prompt Generation

Downloads 31

Release Time : 12/1/2022

Model Overview

This model uses Reinforcement Learning from Human Feedback (RLHF) to fine-tune BLOOM-560m, specifically designed to generate optimized image prompts for Stable Diffusion 2.0. It can automatically expand brief inputs into detailed prompts or independently generate complete prompts.

Model Features

RLHF-Optimized Prompt Quality

Fine-tuned with human feedback reinforcement learning, the generated prompts produce higher-quality images that better align with aesthetic preferences

Stable Diffusion 2.0 Adaptation

Specially optimized for SD2.0, ensuring better compatibility with this version

Lightweight Model

With only 560M parameters, it maintains performance while reducing computational resource requirements

Model Capabilities

Prompt Auto-completion

Complete Prompt Generation

Art Style Description Enhancement

Image Quality Modifier Generation

Use Cases

AI Art Creation

Concept Art Generation

Automatically expands basic themes into detailed art creation prompts

Generates high-quality prompts containing professional descriptions of style, details, lighting, etc.

Prompt Optimization

Professionally optimizes simple prompts provided by users

Enhances output quality by adding art styles, rendering engines, and other professional terminology

🚀 BLOOM-560m RLHF SD2 Prompter

This project uses RLHF (Reinforcement Learning from Human Feedback) to further finetune mrm8488/bloom-560m-finetuned-sd-prompts for SD2.0, aiming to generate better prompts for Stable Diffusion 2.0.

COLAB DEMO INCLUDING STABLE DIFFUSION: https://colab.research.google.com/github/aicrumb/doohickey/blob/main/rlhf_prompt_tuner.ipynb

📄 License

The model is licensed under the bigscience-bloom-rail-1.0 license.

🔖 Tags

stable-diffusion
diffusion

📦 Model Information

Property	Details
Model Name	bloom-560m-RLHF-SD2-prompter
Model Type	Finetuned Transformer for SD2.0 Prompt Generation
Training Data	Gustavosta/Stable-Diffusion-Prompts

🚀 Quick Start

Training Parameters

batch_size = 16
learning_rate = 0.001 # this is why I didn't have to spend _forever_ on it

Generate Prompts

Generate extensions with "<s>Prompt: " followed by your normal prompt.

Donation

Donate so that the developer can run this on real hardware: https://github.com/aicrumb/aicrumb/blob/main/README.md

💻 Usage Examples

Basic Usage

# Install libraries needed to run the models
!pip install transformers diffusers accelerate -qq

# Import the libraries
from diffusers import StableDiffusionPipeline, EulerDiscreteScheduler
from transformers import pipeline
import torch

# This is the model that the transformer was finetuned to generate prompts for
model_id = "stabilityai/stable-diffusion-2-base"

# Use the Euler scheduler here
scheduler = EulerDiscreteScheduler.from_pretrained(model_id, subfolder="scheduler")
pipe = StableDiffusionPipeline.from_pretrained(model_id, scheduler=scheduler, revision="fp16", torch_dtype=torch.float16)
pipe = pipe.to("cuda")

# Load the transformer model
prompt_pipe = pipeline("text-generation", model="crumb/bloom-560m-RLHF-SD2-prompter")
prompt = "cool landscape"

# Auto-complete prompt
prompt = "<s>Prompt: " + prompt + ","
extended_prompt = prompt_pipe(prompt, do_sample=True, max_length=42)[0]['generated_text']
extended_prompt = extended_prompt[10:]
print("Prompt is now: ", extended_prompt)

# Generate image
image = pipe(extended_prompt).images[0]  

image.save("output.png")
image

Example Results

Prompt is now: cool landscape, concept art
Prompt is now: cool landscape, concept art, sharp focus, digital painting

The model can also generate prompts by itself with just the "Prompt:" prompt.

~~Prompt: 1 0 th century, highly detailed, concept art, cinematic lighting, unreal engine, trending on artstation, artstation hd, artstation hq, very very detailed~~

📈 Future Work

Further testing is needed in areas such as automated training with aesthetic predicting models, larger data collection about prompt scores, and better overall training.

Here is a graphic related to the reward methodology:

~~Featured Recommended AI Models~~

Empowering the Future, Your AI Solution Knowledge Base
English 简体中文繁體中文にほんご