Multiview-incontext Open-Source Multi-View Image Generation Model - Synchronously Generate Images of Different Views in the Same Scene

Multiview Incontext

Developed by ysmao

A multi-view image generation model based on FLUX model and LoRA fine-tuning, capable of synchronously generating different views of the same scene

Image Generation Open Source License:MIT #Dual-view synchronous generation #Scene consistency optimization #LoRA fine-tuned diffusion model

Downloads 51

Release Time : 11/21/2024

Model Overview

Inspired by Context LoRA, this model focuses on generating multi-view images of the same scene or object. By combining the FLUX base model with multi-view Context LoRA technology, it can segment images into multiple parts to generate novel views.

Model Features

Dual-view consistency generation

Capable of synchronously generating two different views of the same scene while maintaining consistency of scene elements

LoRA fine-tuning technology

Utilizes LoRA (Low-Rank Adaptation) fine-tuning method to achieve multi-view generation capability while preserving base model performance

Scene understanding and decomposition

Can decompose input text descriptions into multi-view scene descriptions to generate coherent multi-view images

Model Capabilities

Multi-view indoor scene generation

Multi-view object display

Text-based image generation

View consistency maintenance

Use Cases

Interior design

Living room multi-view display

Generate different angle views of a living room for interior design preview

Coherent dual-view images containing elements like sofa, TV, and plants

Bedroom multi-angle presentation

Show layout effects from different bedroom perspectives

Coherent dual-view images containing elements like bed, dressing table, and windows

Product display

Multi-angle object display

Generate standard multi-view images of products

🚀 MultiView-InContext-Lora

This project, inspired by In-Context-LoRA, focuses on generating multi - view images of the same scene or object simultaneously. By integrating flux with the multiview - incontext - lora, it can divide images to obtain novel views.

🚀 Quick Start

Inference

import torch
from diffusers import FluxPipeline

pipeline = FluxPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-dev",
    torch_dtype=torch.bfloat16,
)
pipeline.load_lora_weights(
    "ysmao/multiview-incontext",
    weight_name="twoview-incontext-b03.safetensors",
)
pipeline.fuse_lora()

scene_prompt = "a living room with a sofa set with cushions, side tables with table lamps, a flat screen television on a table, houseplants, wall hangings, electric lights, and a carpet on the floor"
prompt = f"[TWO-VIEWS] This set of two images presents a scene from two different viewpoints. [IMAGE1] The first image shows {scene_prompt}. [IMAGE2] The second image shows the same room but in another viewpoint."
image_height = 576
image_width = 864
output = pipeline(
    prompt=prompt,
    height=int(image_height),
    width=int(image_width * 2),
    num_inference_steps=30,
    guidance_scale=3.5,
).images[0]

output.save("twoview-incontext-beta.png")

✨ Features

Generate multi - view images of the same scene or object simultaneously.
Divide images using flux with the multiview - incontext - lora to obtain novel views.

📚 Documentation

Model description

Inspired by In-Context-LoRA, this project aims to generate multi - view images of the same scene or object simultaneously. By using flux with the multiview - incontext - lora, we can divide the images into portions to obtain novel views.

⚠️ Important Note

This is a beta release of the model. The consistency between views may not be perfect, and the model might sometimes generate views that don't perfectly align or maintain exact object positions across viewpoints. The developer is working on improving the geometric consistency and spatial relationships between generated views.

News

2024 - 11 - 25: Release beta v0.3 model checkpoint, the consistency between views has been improved a lot compared to the previous version.

Roadmap

[x] 🔄 Improve the consistency between the two - view images.
- [ ] Add camera control to the prompt to manage the similarity between the two views.
[ ] 4️⃣ Generate 4 views of a scene in a grid format.
[ ] 🧸 Generate 4 canonical coordinates view points of a single object in a grid format.
[ ] 🏛️ 3D reconstruction from multi - view images.

When applying the LoRA to the FluxInpaint Pipeline, significant degradation in consistency between the generated and input views was noticed. Therefore, the plan is to also train the LoRA for the FluxFill model instead of the original Flux text - to - image model to improve performance.

Download model

Weights for this model are available in Safetensors format.

Download them in the Files & versions tab.

📄 License

This project is licensed under the MIT license.

📦 Installation

No installation steps were provided in the original document, so this section is skipped.

🔧 Technical Details

No specific technical details (more than 50 words) were provided in the original document, so this section is skipped.

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご