VACE Wan2.1 1.3B Preview

🚀 VACE: All-in-One Video Creation and Editing

VACE is an all - in - one model for video creation and editing. It supports multiple tasks like reference - to - video generation, video - to - video editing, and masked video - to - video editing, streamlining workflows and offering diverse capabilities.

🚀 Quick Start

VACE is an all - in - one model designed for video creation and editing. It can handle various tasks such as reference - to - video generation (R2V), video - to - video editing (V2V), and masked video - to - video editing (MV2V). Users can freely combine these tasks to explore different possibilities and simplify their workflows, with capabilities like Move - Anything, Swap - Anything, Reference - Anything, Expand - Anything, Animate - Anything, etc.

✨ Features

• Multi - task Support: Enables reference - to - video generation, video - to - video editing, and masked video - to - video editing.
• Flexible Workflow: Allows users to freely compose different tasks.
• Diverse Capabilities: Offers functions like Move - Anything, Swap - Anything, etc.

📦 Installation

The codebase was tested with Python 3.10.13, CUDA version 12.4, and PyTorch >= 2.5.1.

Setup for Model Inference

You can set up for VACE model inference by running:

git clone https://github.com/ali-vilab/VACE.git && cd VACE
pip install torch==2.5.1 torchvision==0.20.1 --index-url https://download.pytorch.org/whl/cu124  # If PyTorch is not installed.
pip install -r requirements.txt
pip install wan@git+https://github.com/Wan-Video/Wan2.1  # If you want to use Wan2.1-based VACE.
pip install ltx-video@git+https://github.com/Lightricks/LTX-Video@ltx-video-0.9.1 sentencepiece --no-deps # If you want to use LTX-Video-0.9-based VACE. It may conflict with Wan.

Please download your preferred base model to <repo-root>/models/.

Setup for Preprocess Tools

If you need preprocessing tools, please install:

pip install -r requirements/annotator.txt

Please download VACE - Annotators to <repo-root>/models/.

Local Directories Setup

It is recommended to download VACE - Benchmark to <repo-root>/benchmarks/ as examples in run_vace_xxx.sh.

We recommend organizing local directories as:

VACE
├── ...
├── benchmarks
│   └── VACE-Benchmark
│       └── assets
│           └── examples
│               ├── animate_anything
│               │   └── ...
│               └── ...
├── models
│   ├── VACE-Annotators
│   │   └── ...
│   ├── VACE-LTX-Video-0.9
│   │   └── ...
│   └── VACE-Wan2.1-1.3B-Preview
│       └── ...
└── ...

💻 Usage Examples

Basic Usage

In VACE, users can input text prompt and optional video, mask, and image for video generation or editing. Detailed instructions for using VACE can be found in the User Guide.

Advanced Usage

1) End - to - End Running

To simply run VACE without diving into any implementation details, we suggest an end - to - end pipeline. For example:

# run V2V depth
python vace/vace_pipeline.py --base wan --task depth --video assets/videos/test.mp4 --prompt 'xxx'

# run MV2V inpainting by providing bbox
python vace/vace_pipeline.py --base wan --task inpainting --mode bbox --bbox 50,50,550,700 --video assets/videos/test.mp4 --prompt 'xxx'

This script will run video preprocessing and model inference sequentially, and you need to specify all the required args of preprocessing (--task, --mode, --bbox, --video, etc.) and inference (--prompt, etc.). The output video together with intermediate video, mask and images will be saved into ./results/ by default.

⚠️ Important Note Please refer to run_vace_pipeline.sh for usage examples of different task pipelines.

2) Preprocessing

To have more flexible control over the input, before VACE model inference, user inputs need to be preprocessed into src_video, src_mask, and src_ref_images first. We assign each preprocessor a task name, so simply call vace_preprocess.py and specify the task name and task params. For example:

# process video depth
python vace/vace_preproccess.py --task depth --video assets/videos/test.mp4

# process video inpainting by providing bbox
python vace/vace_preproccess.py --task inpainting --mode bbox --bbox 50,50,550,700 --video assets/videos/test.mp4

The outputs will be saved to ./proccessed/ by default.

⚠️ Important Note Please refer to run_vace_pipeline.sh for preprocessing methods for different tasks. Moreover, refer to vace/configs/ for all the pre - defined tasks and required params. You can also customize preprocessors by implementing at annotators and register them at configs.

3) Model inference

Using the input data obtained from Preprocessing, the model inference process can be performed as follows:

# For Wan2.1 single GPU inference 
python vace/vace_wan_inference.py --ckpt_dir <path-to-model> --src_video <path-to-src-video> --src_mask <path-to-src-mask> --src_ref_images <paths-to-src-ref-images> --prompt "xxx"

# For Wan2.1 Multi GPU Acceleration inference
pip install "xfuser>=0.4.1"
torchrun --nproc_per_node=8 vace/vace_wan_inference.py --dit_fsdp --t5_fsdp --ulysses_size 1 --ring_size 8 --ckpt_dir <path-to-model> --src_video <path-to-src-video> --src_mask <path-to-src-mask> --src_ref_images <paths-to-src-ref-images> --prompt "xxx"

# For LTX inference, run
python vace/vace_ltx_inference.py --ckpt_path <path-to-model> --text_encoder_path <path-to-model> --src_video <path-to-src-video> --src_mask <path-to-src-mask> --src_ref_images <paths-to-src-ref-images> --prompt "xxx"

The output video together with intermediate video, mask and images will be saved into ./results/ by default.

⚠️ Important Note (1) Please refer to vace/vace_wan_inference.py and vace/vace_ltx_inference.py for the inference args. (2) For LTX - Video and English language Wan2.1 users, you need prompt extension to unlock the full model performance. Please follow the instruction of Wan2.1 and set --use_prompt_extend while running inference.

Inference Gradio

For preprocessors, run

python vace/gradios/preprocess_demo.py

For model inference, run

# For Wan2.1 gradio inference
python vace/gradios/vace_wan_demo.py

# For LTX gradio inference
python vace/gradios/vace_ltx_demo.py

📚 Documentation

Models

• The input supports any resolution, but to achieve optimal results, the video size should fall within a specific range.
• All models inherit the license of the original model.

🎉 News

• [x] Mar 31, 2025: 🔥VACE - Wan2.1 - 1.3B - Preview and VACE - LTX - Video - 0.9 models are now available at HuggingFace and ModelScope!
• [x] Mar 31, 2025: 🔥Release code of model inference, preprocessing, and gradio demos.
• [x] Mar 11, 2025: We propose VACE, an all - in - one model for video creation and editing.

Acknowledgement

We are grateful for the following awesome projects, including Scepter, Wan, and LTX - Video.

📄 License

The models are released under the Apache - 2.0 or RAIL - M license.

BibTeX

@article{vace,
    title = {VACE: All-in-One Video Creation and Editing},
    author = {Jiang, Zeyinzi and Han, Zhen and Mao, Chaojie and Zhang, Jingfeng and Pan, Yulin and Liu, Yu},
    journal = {arXiv preprint arXiv:2503.07598},
    year = {2025}
}