Segmoe 2x1 V0

🚀 SegMoE-2x1-v0: Segmind Mixture of Diffusion Experts

SegMoE-2x1-v0 is an untrained Segmind Mixture of Diffusion Experts Model. It leverages the segmoe framework to combine 2 Expert SDXL models, enabling the dynamic creation of larger models with enhanced knowledge, better adherence, and superior image quality.

🚀 Quick Start

SegMoE-2x1-v0 can be used via the segmoe library. First, install the segmoe library:

pip install segmoe

💻 Usage Examples

Basic Usage

from segmoe import SegMoEPipeline

pipeline = SegMoEPipeline("segmind/SegMoE-2x1-v0", device = "cuda")

prompt = "cosmic canvas, orange city background, painting of a chubby cat"
negative_prompt = "nsfw, bad quality, worse quality"
img = pipeline(
    prompt=prompt,
    negative_prompt=negative_prompt,
    height=1024,
    width=1024,
    num_inference_steps=25,
    guidance_scale=7.5,
).images[0]
img.save("image.png")

Config

The configuration used to create this model is as follows:

base_model: SG161222/RealVisXL_V3.0
num_experts: 2
moe_layers: all
num_experts_per_tok: 1
experts:
  - source_model: frankjoshua/juggernautXL_v8Rundiffusion
    positive_prompt: "aesthetic, cinematic, hands, portrait, photo, illustration, 8K, hyperdetailed, origami, man, woman, supercar"
    negative_prompt: "(worst quality, low quality, normal quality, lowres, low details, oversaturated, undersaturated, overexposed, underexposed, grayscale, bw, bad photo, bad photography, bad art:1.4), (watermark, signature, text font, username, error, logo, words, letters, digits, autograph, trademark, name:1.2), (blur, blurry, grainy), morbid, ugly, asymmetrical, mutated malformed, mutilated, poorly lit, bad shadow, draft, cropped, out of frame, cut off, censored, jpeg artifacts, out of focus, glitch, duplicate, (airbrushed, cartoon, anime, semi-realistic, cgi, render, blender, digital art, manga, amateur:1.3), (3D ,3D Game, 3D Game Scene, 3D Character:1.1), (bad hands, bad anatomy, bad body, bad face, bad teeth, bad arms, bad legs, deformities:1.3)"
  - source_model: SG161222/RealVisXL_V3.0
    positive_prompt: "cinematic, portrait, photograph, instagram, fashion, movie, macro shot, 8K, RAW, hyperrealistic, ultra realistic,"
    negative_prompt: "(octane render, render, drawing, anime, bad photo, bad photography:1.3), (worst quality, low quality, blurry:1.2), (bad teeth, deformed teeth, deformed lips), (bad anatomy, bad proportions:1.1), (deformed iris, deformed pupils), (deformed eyes, bad eyes), (deformed face, ugly face, bad face), (deformed hands, bad hands, fused fingers), morbid, mutilated, mutation, disfigured"

Other Variants

We have released 3 merged models on Hugging Face:

• SegMoE 4x2: This model uses four expert models.
• SegMoE SD 4x2: It consists of four Stable Diffusion 1.5 expert models.

📚 Documentation

Model Comparison

The prompt understanding of SegMoE models seems to improve, as shown in the following images. From left to right: SegMoE-2x1-v0, SegMoE-4x2-v0, and the base model (RealVisXL_V3.0).

Model Description

Out-of-Scope Use

The SegMoE-2x1-v0 model is not suitable for creating factual or accurate representations of people, events, or real-world information. It is not intended for tasks requiring high precision and accuracy.

✨ Features

• Leverage Expert Knowledge: Benefit from the knowledge of multiple fine-tuned experts.
• Training-Free: No need for additional training.
• Better Adaptability: Adapt well to different data.
• Model Upgrade: The model can be upgraded by using a better fine-tuned model as one of the experts.

🔧 Technical Details

Limitations

• Dependence on Experts: Although the model improves image fidelity and adherence, it does not significantly outperform any single expert without training and relies on the knowledge of the experts.
• Speed Optimization: The model is not yet optimized for speed.
• Memory Usage: The framework is not yet optimized for memory usage.

📄 License

This model is licensed under the Apache 2.0 license.

📚 Citation

@misc{segmoe,
  author = {Yatharth Gupta, Vishnu V Jaddipal, Harish Prabhala},
  title = {SegMoE},
  year = {2024},
  publisher = {HuggingFace},
  journal = {HuggingFace Models},
  howpublished = {\url{https://huggingface.co/segmind/SegMoE-2x1-v0}}
}