Ruyi Mini 7B

🚀 Ruyi-Mini-7B

Ruyi-Mini-7B is an open - source image - to - video generation model developed by CreateAI. It can generate video frames from an input image, with resolutions from 360p to 720p, supporting various aspect ratios and a maximum duration of 5 seconds. The model offers greater flexibility and creativity in video generation with motion and camera control.

Hugging Face | Github

🚀 Quick Start

Ruyi-Mini-7B is an open - source image - to - video generation model. Starting with an input image, it can produce subsequent video frames at resolutions ranging from 360p to 720p, supporting various aspect ratios and a maximum duration of 5 seconds.

✨ Features

• Generate video frames from an input image with resolutions from 360p to 720p.
• Support various aspect ratios and a maximum video duration of 5 seconds.
• Enhanced with motion and camera control for more flexible and creative video generation.

📦 Installation

Install code from github:

git clone https://github.com/IamCreateAI/Ruyi-Models
cd Ruyi-Models
pip install -r requirements.txt

💻 Usage Examples

Basic Usage

We provide two ways to run our model. The first is directly using python code.

python3 predict_i2v.py

Advanced Usage

Or use ComfyUI wrapper in our github repo.

📚 Documentation

Model Architecture

Ruyi-Mini-7B is an advanced image - to - video model with about 7.1 billion parameters. The model architecture is modified from [EasyAnimate V4 model](https://github.com/aigc - apps/EasyAnimate), whose transformer module is inherited from HunyuanDiT. It comprises three key components:

1. Casual VAE Module: Handles video compression and decompression. It reduces spatial resolution to 1/8 and temporal resolution to 1/4, with each latent pixel is represented in 16 float numbers after compression.
2. Diffusion Transformer Module: Generates compressed video data using 3D full attention, with:
   • 2D Normalized - RoPE for spatial dimensions;
   • Sin - cos position embedding for temporal dimensions;
   • DDPM (Denoising Diffusion Probabilistic Models) for model training.
3. Ruyi also utilizes a CLIP model to extract the semantic features from the input image to guide the whole video generation. The CLIP features are introduced into the transformer by cross - attention.

Training Data and Methodology

The training process is divided into four phases:

• Phase 1: Pre - training from scratch with ~200M video clips and ~30M images at a 256 - resolution, using a batch size of 4096 for 350,000 iterations to achieve full convergence.
• Phase 2: Fine - tuning with ~60M video clips for multi - scale resolutions (384–512), with a batch size of 1024 for 60,000 iterations.
• Phase 3: High - quality fine - tuning with ~20M video clips and ~8M images for 384–1024 resolutions, with dynamic batch sizes based on memory and 10,000 iterations.
• Phase 4: Image - to - video training with ~10M curated high - quality video clips, with dynamic batch sizes based on memory for ~10,000 iterations.

Hardware Requirements

The VRAM cost of Ruyi depends on the resolution and duration of the video. Here we list the costs for some typical video size. Tested on single A100.

For 24GB VRAM cards such as RTX4090, we provide low_gpu_memory_mode, under which the model can generate 720x1280x120 videos with a longer time.

Showcase

Image to Video Effects

Camera Control

input	left	right
static	up	down

Motion Amplitude Control

Limitations

There are some known limitations in this experimental release. Texts, hands and crowded human faces may be distorted. The video may cut to another scene when the model does not know how to generate future frames. We are still working on these problems and will update the model as we make progress.

Update

• Dec 24, 2024: The diffusion model is updated to fix the black lines when creating 3:4 or 4:5 videos.
• Dec 16, 2024: Ruyi - mini - 7B is released.

🔧 Technical Details

The model architecture and training details are described in the "Model Architecture" and "Training Data and Methodology" sections above.

📄 License

This model is released under the permissive Apache 2.0 license.

BibTeX

@misc{createai2024ruyi,
      title={Ruyi-Mini-7B},
      author={CreateAI Team},
      year={2024},
      publisher = {GitHub},
      journal = {GitHub repository},
      howpublished={\url{https://github.com/IamCreateAI/Ruyi-Models}}
}

Contact Us

You are welcomed to join our Discord or Wechat Group (Scan QR code to add Ruyi Assistant and join the official group) for further discussion!