Skyreels V2 DF 14B 720P

🚀 SkyReels V2: Infinite-Length Film Generative Model

SkyReels V2 is an infinite-length film generative model. It's the first open - source video generative model using AutoRegressive Diffusion - Forcing architecture, achieving SOTA performance among publicly available models.

✨ Features

• Innovative Architecture: Employs AutoRegressive Diffusion - Forcing architecture.
• Multiple Model Variants: Offers different model sizes and resolutions for various needs.
• Diverse Applications: Supports story generation, image - to - video synthesis, camera director functionality, etc.

📦 Installation

# clone the repository.
git clone https://github.com/SkyworkAI/SkyReels-V2
cd SkyReels-V2
# Install dependencies. Test environment uses Python 3.10.12.
pip install -r requirements.txt

💻 Usage Examples

Basic Usage

Model Download

You can download our models from Hugging Face:

After downloading, set the model path in your generation commands.

Advanced Usage

Single GPU Inference

• Diffusion Forcing for Long Video Generation The Diffusion Forcing version model allows us to generate Infinite - Length videos. This model supports both text - to - video (T2V) and image - to - video (I2V) tasks, and it can perform inference in both synchronous and asynchronous modes.

Synchronous generation for 10s video

model_id=Skywork/SkyReels-V2-DF-14B-540P
# synchronous inference
python3 generate_video_df.py \
  --model_id ${model_id} \
  --resolution 540P \
  --ar_step 0 \
  --base_num_frames 97 \
  --num_frames 257 \
  --overlap_history 17 \
  --prompt "A graceful white swan with a curved neck and delicate feathers swimming in a serene lake at dawn, its reflection perfectly mirrored in the still water as mist rises from the surface, with the swan occasionally dipping its head into the water to feed." \
  --addnoise_condition 20 \
  --offload \
  --teacache \
  --use_ret_steps \
  --teacache_thresh 0.3

Asynchronous generation for 30s video

model_id=Skywork/SkyReels-V2-DF-14B-540P
# asynchronous inference
python3 generate_video_df.py \
  --model_id ${model_id} \
  --resolution 540P \
  --ar_step 5 \
  --causal_block_size 5 \
  --base_num_frames 97 \
  --num_frames 737 \
  --overlap_history 17 \
  --prompt "A graceful white swan with a curved neck and delicate feathers swimming in a serene lake at dawn, its reflection perfectly mirrored in the still water as mist rises from the surface, with the swan occasionally dipping its head into the water to feed." \
  --addnoise_condition 20 \
  --offload

⚠️ Important Note

• If you want to run the image - to - video (I2V) task, add --image ${image_path} to your command and it is also better to use text - to - video (T2V) - like prompt which includes some descriptions of the first - frame image.
• For long video generation, you can just switch the --num_frames, e.g., --num_frames 257 for 10s video, --num_frames 377 for 15s video, --num_frames 737 for 30s video, --num_frames 1457 for 60s video. The number is not strictly aligned with the logical frame number for specified time duration, but it is aligned with some training parameters, which means it may perform better. When you use asynchronous inference with causal_block_size > 1, the --num_frames should be carefully set.
• You can use --ar_step 5 to enable asynchronous inference. When asynchronous inference, --causal_block_size 5 is recommended while it is not supposed to be set for synchronous generation. REMEMBER that the frame latent number inputted into the model in every iteration, e.g., base frame latent number (e.g., (97 - 1)//4+1 = 25 for base_num_frames = 97) and (e.g., (237 - 97-(97 - 17)x1+17 - 1)//4+1 = 20 for base_num_frames = 97, num_frames = 237, overlap_history = 17) for the last iteration, MUST be divided by causal_block_size. If you find it too hard to calculate and set proper values, just use our recommended setting above :). Asynchronous inference will take more steps to diffuse the whole sequence which means it will be SLOWER than synchronous mode. In our experiments, asynchronous inference may improve the instruction following and visual consistent performance.
• To reduce peak VRAM, just lower the --base_num_frames, e.g., to 77 or 57, while keeping the same generative length --num_frames you want to generate. This may slightly reduce video quality, and it should not be set too small.
• --addnoise_condition is used to help smooth the long video generation by adding some noise to the clean condition. Too large noise can cause the inconsistency as well. 20 is a recommended value, and you may try larger ones, but it is recommended to not exceed 50.
• Generating a 540P video using the 1.3B model requires approximately 14.7GB peak VRAM, while the same resolution video using the 14B model demands around 51.2GB peak VRAM.

• Text To Video & Image To Video

# run Text - to - Video Generation
model_id=Skywork/SkyReels-V2-T2V-14B-540P
python3 generate_video.py \
  --model_id ${model_id} \
  --resolution 540P \
  --num_frames 97 \
  --guidance_scale 6.0 \
  --shift 8.0 \
  --fps 24 \
  --prompt "A serene lake surrounded by towering mountains, with a few swans gracefully gliding across the water and sunlight dancing on the surface." \
  --offload \
  --teacache \
  --use_ret_steps \
  --teacache_thresh 0.3

⚠️ Important Note

• When using an image - to - video (I2V) model, you must provide an input image using the --image ${image_path} parameter. The --guidance_scale 5.0 and --shift 3.0 is recommended for I2V model.
• Generating a 540P video using the 1.3B model requires approximately 14.7GB peak VRAM, while the same resolution video using the 14B model demands around 43.4GB peak VRAM.

• Prompt Enhancer The prompt enhancer is implemented based on Qwen2.5 - 32B - Instruct and is utilized via the --prompt_enhancer parameter. It works ideally for short prompts, while for long prompts, it might generate an excessively lengthy prompt that could lead to over - saturation in the generative video. Note the peak memory of GPU is 64G+ if you use --prompt_enhancer. If you want to obtain the enhanced prompt separately, you can also run the prompt_enhancer script separately for testing. The steps are as follows:

cd skyreels_v2_infer/pipelines
python3 prompt_enhancer.py --prompt "A serene lake surrounded by towering mountains, with a few swans gracefully gliding across the water and sunlight dancing on the surface."

⚠️ Important Note

• --prompt_enhancer is not allowed if using --use_usp. We recommend running the skyreels_v2_infer/pipelines/prompt_enhancer.py script first to generate enhanced prompt before enabling the --use_usp parameter.

Multi - GPU inference using xDiT USP

We use xDiT USP to accelerate inference. For example, to generate a video with 2 GPUs, you can use the following command:

• Diffusion Forcing

model_id=Skywork/SkyReels-V2-DF-14B-540P
# diffusion forcing synchronous inference
torchrun --nproc_per_node=2 generate_video_df.py \
  --model_id ${model_id} \
  --resolution 540P \
  --ar_step 0 \
  --base_num_frames 97 \
  --num_frames 257 \
  --overlap_history 17 \
  --prompt "A graceful white swan with a curved neck and delicate feathers swimming in a serene lake at dawn, its reflection perfectly mirrored in the still water as mist rises from the surface, with the swan occasionally dipping its head into the water to feed." \
  --addnoise_condition 20 \
  --use_usp \
  --offload \
  --seed 42

• Text To Video & Image To Video

# run Text - to - Video Generation
model_id=Skywork/SkyReels-V2-T2V-14B-540P
torchrun --nproc_per_node=2 generate_video.py \
  --model_id ${model_id} \
  --resolution 540P \
  --num_frames 97 \
  --guidance_scale 6.0 \
  --shift 8.0 \
  --fps 24 \
  --offload \
  --prompt "A serene lake surrounded by towering mountains, with a few swans gracefully gliding across the water and sunlight dancing on the surface." \
  --use_usp \
  --seed 42

⚠️ Important Note

• When using an image - to - video (I2V) model, you must provide an input image using the --image ${image_path} parameter. The --guidance_scale 5.0 and --shift 3.0 is recommended for I2V model.

📚 Documentation

Abstract

Recent advances in video generation have been driven by diffusion models and autoregressive frameworks, yet critical challenges persist in harmonizing prompt adherence, visual quality, motion dynamics, and duration: compromises in motion dynamics to enhance temporal visual quality, constrained video duration (5 - 10 seconds) to prioritize resolution, and inadequate shot - aware generation stemming from general - purpose MLLMs' inability to interpret cinematic grammar, such as shot composition, actor expressions, and camera motions. These intertwined limitations hinder realistic long - form synthesis and professional film - style generation.

To address these limitations, we introduce SkyReels - V2, the world's first infinite - length film generative model using a Diffusion Forcing framework. Our approach synergizes Multi - modal Large Language Models (MLLM), Multi - stage Pretraining, Reinforcement Learning, and Diffusion Forcing techniques to achieve comprehensive optimization. Beyond its technical innovations, SkyReels - V2 enables multiple practical applications, including Story Generation, Image - to - Video Synthesis, Camera Director functionality, and multi - subject consistent video generation through our Skyreels - A2 system.

Methodology of SkyReels - V2

The SkyReels - V2 methodology consists of several interconnected components. It starts with a comprehensive data processing pipeline that prepares various quality training data. At its core is the Video Captioner architecture, which provides detailed annotations for video content. The system employs a multi - task pretraining strategy to build fundamental video generation capabilities. Post - training optimization includes Reinforcement Learning to enhance motion quality, Diffusion Forcing Training for generating extended videos, and High - quality Supervised Fine - Tuning (SFT) stages for visual refinement. The model runs on optimized computational infrastructure for efficient training and inference. SkyReels - V2 supports multiple applications, including Story Generation, Image - to - Video Synthesis, Camera Director functionality, and Elements - to - Video Generation.

Key Contributions of SkyReels - V2

Video Captioner

SkyCaptioner - V1 serves as our video captioner... (The rest of the content about Video Captioner should be continued here if there is more)

Performance

(There is no specific content about performance in the original text. If there is more information later, it should be added here.)

Acknowledgements

(There is no specific content about acknowledgements in the original text. If there is more information later, it should be added here.)

Citation

(There is no specific content about citation in the original text. If there is more information later, it should be added here.)

🔧 Technical Details

Advanced Configuration Options Below are the key parameters you can customize for video generation:

Diffusion Forcing Additional Parameters

📄 License

This project is under the skywork - license.