Genex World Explorer

🚀 GenEx-World-Explorer

GenEx World Explorer is a video generation pipeline built on Stable Video Diffusion (SVD). It takes a keyframe and generates a temporally consistent video. This explorer version enhances SVD with a custom UNetSpatioTemporalConditionModel. The diffuser generates a forward - moving path for a panoramic input image to explore a given scene.

🚀 Quick Start

GenEx World Explorer is a powerful video generation pipeline. It leverages the Stable Video Diffusion framework and a custom model to create temporally - consistent videos from keyframes.

✨ Features

• Base Model: Built on Stable Video Diffusion (SVD).
• Custom Model: Utilizes a custom UNetSpatioTemporalConditionModel for enhanced performance.
• Functionality: Generates a forward - moving path for panoramic input images to explore scenes.

📦 Installation

The following libraries are required to use GenEx World Explorer:

diffusers>=0.33.1
transformers
numpy
pillow

💻 Usage Examples

Basic Usage

from diffusers import UNetSpatioTemporalConditionModel, StableVideoDiffusionPipeline
import torch
from PIL import Image

model_id = 'genex-world/GenEx-World-Explorer'

# Load the custom UNet
unet = UNetSpatioTemporalConditionModel.from_pretrained(
    model_id,
    subfolder='unet',
    torch_dtype=torch.float16,
    low_cpu_mem_usage=True,
)

# Load the full pipeline with custom UNet
pipe = StableVideoDiffusionPipeline.from_pretrained(
    model_id,
    unet=unet,
    low_cpu_mem_usage=True,
    torch_dtype=torch.float16,
    local_files_only=True,
).to('cuda')

# Explore the world!
image = Image.open('example.png').resize((1024, 576), Image.BICUBIC).convert('RGB')

generator = torch.manual_seed(-1)
with torch.inference_mode():
    frames = pipe(image,
                num_frames=25,
                width=1024,
                height=576,
                decode_chunk_size=8, generator=generator, motion_bucket_id=127, fps=7, num_inference_steps=30, noise_aug_strength=0.02).frames[0]

📄 License

This project is licensed under the cc - by - 4.0 license.

✨ BibTex

@misc{lu2025genexgeneratingexplorableworld,
      title={GenEx: Generating an Explorable World}, 
      author={Taiming Lu and Tianmin Shu and Junfei Xiao and Luoxin Ye and Jiahao Wang and Cheng Peng and Chen Wei and Daniel Khashabi and Rama Chellappa and Alan Yuille and Jieneng Chen},
      year={2025},
      eprint={2412.09624},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2412.09624}, 
}