Sotediffusion Wuerstchen3

🚀 SoteDiffusion Wuerstchen3

Anime finetune of Würstchen V3. This model offers high - quality anime image generation capabilities, fine - tuned to meet specific aesthetic and quality standards.

🚀 Quick Start

A new version is available: https://huggingface.co/Disty0/sotediffusion-v2

✨ Features

• This release is sponsored by fal.ai/grants.
• Trained on 6M images for 3 epochs using 8x A100 80G GPUs.

💻 Usage Examples

API Usage

This model can be used via API with Fal.AI. For more details: https://fal.ai/models/fal-ai/stable-cascade/sote-diffusion

UI Guide

SD.Next

URL: https://github.com/vladmandic/automatic/

Go to Models -> Huggingface and type Disty0/sotediffusion-wuerstchen3-decoder into the model name and press download. Load Disty0/sotediffusion-wuerstchen3-decoder after the download process is complete.

Prompt:

newest, extremely aesthetic, best quality,

Negative Prompt:

very displeasing, worst quality, monochrome, realistic, oldest, loli,

Parameters: Sampler: Default Steps: 30 or 40 Refiner Steps: 10 CFG: 7 Secondary CFG: 2 or 1 Resolution: 1024x1536, 2048x1152. Anything works as long as it's a multiply of 128.

ComfyUI

Please refer to CivitAI: https://civitai.com/models/353284

Code Example

Basic Usage

import torch
from diffusers import StableCascadeCombinedPipeline

device = "cuda"
dtype = torch.bfloat16 # or torch.float16
model = "Disty0/sotediffusion-wuerstchen3-decoder"

pipe = StableCascadeCombinedPipeline.from_pretrained(model, torch_dtype=dtype)

# send everything to the gpu:
pipe = pipe.to(device, dtype=dtype)
pipe.prior_pipe = pipe.prior_pipe.to(device, dtype=dtype)

# or enable model offload to save vram:
# pipe.enable_model_cpu_offload()

prompt = "newest, extremely aesthetic, best quality, 1girl, solo, cat ears, pink hair, orange eyes, long hair, bare shoulders, looking at viewer, smile, indoors, casual, living room, playing guitar,"
negative_prompt = "very displeasing, worst quality, monochrome, realistic, oldest, loli,"
output = pipe(
    width=1024,
    height=1536,
    prompt=prompt,
    negative_prompt=negative_prompt,
    decoder_guidance_scale=2.0,
    prior_guidance_scale=7.0,
    prior_num_inference_steps=30,
    output_type="pil",
    num_inference_steps=10
).images[0]

## do something with the output image

🔧 Technical Details

Training

Software used: Kohya SD - Scripts with Stable Cascade branch. https://github.com/kohya-ss/sd-scripts/tree/stable-cascade GPU used: 8x Nvidia A100 80GB GPU Hours: 220

Base

Final

Dataset

GPU used for captioning: 1x Intel ARC A770 16GB GPU Hours: 350 Model used for captioning: SmilingWolf/wd - swinv2 - tagger - v3 Model used for text: llava - hf/llava - 1.5 - 7b - hf

Command:

python /mnt/DataSSD/AI/Apps/kohya_ss/sd-scripts/finetune/tag_images_by_wd14_tagger.py --model_dir "/mnt/DataSSD/AI/models/wd14_tagger_model" --repo_id "SmilingWolf/wd-swinv2-tagger-v3" --recursive --remove_underscore --use_rating_tags --character_tags_first --character_tag_expand --append_tags --onnx --caption_separator ", " --general_threshold 0.35 --character_threshold 0.50 --batch_size 4 --caption_extension ".txt" ./

Note:

• Smallest size is 1280x600 (768,000 pixels)
• Deduped based on image similarity using czkawka - cli
• Around 120K very high quality images got intentionally duplicated 5 times, making the total image count 6.2M

Tags

Model is trained with random tag order but this is the order in the dataset if you are interested:

aesthetic tags, quality tags, date tags, custom tags, rating tags, character, series, rest of the tags

Date

Aesthetic Tags

Model used: shadowlilac/aesthetic - shadow - v2

Quality Tags

Model used: https://huggingface.co/hakurei/waifu - diffusion - v1 - 4/blob/main/models/aes - B32 - v0.pth

Rating Tags

Custom Tags

Limitations and Bias

Bias

• This model is intended for anime illustrations. Realistic capabilites are not tested at all.

Limitations

• Can fall back to realistic. Add "realistic" tag to the negatives when this happens.
• Far shot eyes and hands can be bad.

📄 License

SoteDiffusion models falls under Fair AI Public License 1.0 - SD license, which is compatible with Stable Diffusion models’ license. Key points:

1. Modification Sharing: If you modify SoteDiffusion models, you must share both your changes and the original license.
2. Source Code Accessibility: If your modified version is network - accessible, provide a way (like a download link) for others to get the source code. This applies to derived models too.
3. Distribution Terms: Any distribution must be under this license or another with similar rules.
4. Compliance: Non - compliance must be fixed within 30 days to avoid license termination, emphasizing transparency and adherence to open - source values.

Notes: Anything not covered by Fair AI license is inherited from Stability AI Non - Commercial license which is named as LICENSE_INHERIT.