Text2motion

🚀 Text2Motion

Text2Motion: Open and Advanced Large-Scale Video Generative Models. It's a comprehensive and open suite of video foundation models that breaks the limits of video generation.

🚀 Quick Start

In this repository, we present Text2Motion, a comprehensive and open suite of video foundation models that pushes the boundaries of video generation.

Installation

First, clone the repository:

git clone https://huggingface.co/sbapan41/Text2Motion
cd Text2Motion

Then, install the required dependencies:

# Ensure torch >= 2.4.0
pip install -r requirements.txt

Model Download

You can download the models using the following table and commands:

Use the huggingface-cli to download the model:

pip install "huggingface_hub[cli]"
huggingface-cli download sbapan41/Text2Motion --local-dir ./Text2Motion

Run Text - to - Video Generation

This repository supports two Text - to - Video models (14B) and two resolutions (480P and 720P). The following table shows the details:

(1) Without Prompt Extention

We'll start with a basic inference process that skips the prompt extension step.

• Single - GPU inference

python generate.py  --task 14B --size 1280*720 --ckpt_dir ./Text2Motion --prompt "Two anthropomorphic cats in comfy boxing gear and bright gloves fight intensely on a spotlighted stage."

If you encounter OOM (Out - of - Memory) issues, you can use the --offload_model True and --t5_cpu options to reduce GPU memory usage. For example, on an RTX 4090 GPU.

• Multi - GPU inference using FSDP + xDiT USP

pip install "xfuser>=0.4.1"
torchrun --nproc_per_node=8 generate.py --task 14B --size 1280*720 --ckpt_dir ./Text2Motion --dit_fsdp --t5_fsdp --ulysses_size 8 --prompt "Two anthropomorphic cats in comfy boxing gear and bright gloves fight intensely on a spotlighted stage."

✨ Features

• 👍 SOTA Performance: Text2Motion consistently outperforms existing open - source models and state - of - the - art commercial solutions across multiple benchmarks.
• 👍 Supports Consumer - grade GPUs: The T2V - 1.3B model requires only 8.19 GB VRAM, making it compatible with almost all consumer - grade GPUs. It can generate a 5 - second 480P video on an RTX 4090 in about 4 minutes (without optimization techniques like quantization). Its performance is even comparable to some closed - source models.
• 👍 Multiple Tasks: Text2Motion excels in Text - to - Video, Image - to - Video, Video Editing, Text - to - Image, and Video - to - Audio, advancing the field of video generation.
• 👍 Visual Text Generation: Text2Motion is the first video model capable of generating both Chinese and English text, featuring robust text generation that enhances its practical applications.
• 👍 Powerful Video VAE: Text2Motion - VAE delivers exceptional efficiency and performance, encoding and decoding 1080P videos of any length while preserving temporal information, making it an ideal foundation for video and image generation.

This repository features our T2V - 14B model, which establishes a new SOTA performance benchmark among both open - source and closed - source models. It demonstrates exceptional capabilities in generating high - quality visuals with significant motion dynamics. It is also the only video model capable of producing both Chinese and English text and supports video generation at both 480P and 720P resolutions.

📚 Documentation

🔥 Latest News!!

• Feb 22, 2025: 👋 We've released the inference code and weights of Text2Motion.

📑 Todo List

• Text2Motion Text - to - Video
  • [x] Multi - GPU Inference code of the 14B
  • [x] Checkpoints of the 14B
  • [x] Gradio demo
  • [ ] Diffusers integration
  • [ ] ComfyUI integration
• Text2Motion Image - to - Video
  • [x] Multi - GPU Inference code of the 14B model
  • [x] Checkpoints of the 14B model
  • [x] Gradio demo
  • [ ] Diffusers integration
  • [ ] ComfyUI integration

🔧 Technical Details

📄 License

This project is licensed under the Apache - 2.0 license.