Segmoe SD 4x2 V0

🚀 SegMoE-SD-4x2-v0: Segmind Mixture of Diffusion Experts

SegMoE-SD-4x2-v0 is an untrained Segmind Mixture of Diffusion Experts Model. It's generated using segmoe from 4 Expert SD1.5 models. SegMoE is a powerful framework that can dynamically combine Stable Diffusion Models into a Mixture of Experts within minutes without training. This framework enables the on - the - fly creation of larger models, offering broader knowledge, better adherence, and superior image quality.

🚀 Quick Start

This model can be used via the segmoe library. First, make sure to install segmoe by running the following command:

pip install segmoe

💻 Usage Examples

Basic Usage

from segmoe import SegMoEPipeline

pipeline = SegMoEPipeline("segmind/SegMoE-SD-4x2-v0", device = "cuda")

prompt = "cosmic canvas, orange city background, painting of a chubby cat"
negative_prompt = "nsfw, bad quality, worse quality"
img = pipeline(
    prompt=prompt,
    negative_prompt=negative_prompt,
    height=1024,
    width=1024,
    num_inference_steps=25,
    guidance_scale=7.5,
).images[0]
img.save("image.png")

Config

The config used to create this model is as follows:

base_model: SG161222/Realistic_Vision_V6.0_B1_noVAE
num_experts: 4
moe_layers: all
num_experts_per_tok: 2
experts:
  - source_model: SG161222/Realistic_Vision_V6.0_B1_noVAE
    positive_prompt: "cinematic, portrait, photograph, instagram, fashion, movie, macro shot, 8K, RAW, hyperrealistic, ultra realistic,"
    negative_prompt: " (deformed iris, deformed pupils, semi-realistic, cgi, 3d, render, sketch, cartoon, drawing, anime), text, cropped, out of frame, worst quality, low quality, jpeg artifacts, ugly, duplicate, morbid, mutilated, extra fingers, mutated hands, poorly drawn hands, poorly drawn face, mutation, deformed, blurry, dehydrated, bad anatomy, bad proportions, extra limbs, cloned face, disfigured, gross proportions, malformed limbs, missing arms, missing legs, extra arms, extra legs, fused fingers, too many fingers, long neck"
  - source_model: dreamlike-art/dreamlike-anime-1.0
    positive_prompt: "photo anime, masterpiece, high quality, absurdres, 1girl, 1boy, waifu, chibi"
    negative_prompt: "simple background, duplicate, retro style, low quality, lowest quality, 1980s, 1990s, 2000s, 2005 2006 2007 2008 2009 2010 2011 2012 2013, bad anatomy, bad proportions, extra digits, lowres, username, artist name, error, duplicate, watermark, signature, text, extra digit, fewer digits, worst quality, jpeg artifacts, blurry"
  - source_model: Lykon/dreamshaper-8
    positive_prompt: "bokeh, intricate, elegant, sharp focus, soft lighting, vibrant colors, dreamlike, fantasy, artstation, concept art"
    negative_prompt: "low quality, lowres, jpeg artifacts, signature, bad anatomy, extra legs, extra arms, extra fingers, poorly drawn hands, poorly drawn feet, disfigured, out of frame, tiling, bad art, deformed, mutated, blurry, fuzzy, misshaped, mutant, gross, disgusting, ugly, watermark, watermarks"
  - source_model: dreamlike-art/dreamlike-diffusion-1.0
    positive_prompt: "dreamlikeart, a grungy woman with rainbow hair, travelling between dimensions, dynamic pose, happy, soft eyes and narrow chin, extreme bokeh, dainty figure, long hair straight down, torn kawaii shirt and baggy jeans, In style of by Jordan Grimmer and greg rutkowski, crisp lines and color, complex background, particles, lines, wind, concept art, sharp focus, vivid colors"
    negative_prompt: "nude, naked, low quality, lowres, jpeg artifacts, signature, bad anatomy, extra legs, extra arms, extra fingers, poorly drawn hands, poorly drawn feet, disfigured, out of frame"

Other Variants

We have released 3 merges on Hugging Face:

• SegMoE 4x2 features four Stable Diffusion XL expert models.
• SegMoE 2x1 consists of two Stable Diffusion XL expert models.

📚 Documentation

Model Description

Out-of-Scope Use

The SegMoE-SD-4x2-v0 Model is not suitable for creating factual or accurate representations of people, events, or real-world information. It is not intended for tasks requiring high precision and accuracy.

✨ Features

• Benefits from the knowledge of several finetuned experts.
• Training-free.
• Better adaptability to data.
• The model can be upgraded by using a better finetuned model as one of the experts.

🔧 Technical Details

Limitations

• Although the model improves the fidelity and adherence of images, it doesn't significantly outperform any single expert without training and relies on the knowledge of the experts.
• It is not yet optimized for speed.
• The framework is not yet optimized for memory usage.

📄 License

The model is released under the Apache 2.0 license.

📖 Citation

@misc{segmoe,
  author = {Yatharth Gupta, Vishnu V Jaddipal, Harish Prabhala},
  title = {SegMoE},
  year = {2024},
  publisher = {HuggingFace},
  journal = {HuggingFace Models},
  howpublished = {\url{https://huggingface.co/segmind/SegMoE-SD-4x2-v0}}
}