Vidxtend

🚀 VidXTend - StreamingT2V Stage 2 Pipeline

This repository offers a pruned and isolated pipeline named "VidXTend" for Stage 2 of StreamingT2V. Its main function is to extend 16 - frame 256px x 256px animations by 8 frames at a time (equivalent to one second at 8fps).

📄 License

This project is licensed under the MIT license.

📚 Citation

@article{henschel2024streamingt2v,
  title={StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text},
  author={Henschel, Roberto and Khachatryan, Levon and Hayrapetyan, Daniil and Poghosyan, Hayk and Tadevosyan, Vahram and Wang, Zhangyang and Navasardyan, Shant and Shi, Humphrey},
  journal={arXiv preprint arXiv:2403.14773},
  year={2024}
}

🔗 Code Repository

Code: https://github.com/Picsart-AI-Research/StreamingT2V

🚀 Quick Start

📦 Installation

First, install the VidXTend package into your Python environment. If you're creating a new environment for VidXTend, make sure to specify the version of torch with CUDA support; otherwise, it will run only on the CPU.

pip install git+https://github.com/painebenjamin/vidxtend.git

💻 Usage Examples

🖥️ Command - Line

A command - line utility vidxtend is installed with the package.

Usage: vidxtend [OPTIONS] VIDEO PROMPT

  Run VidXtend on a video file, concatenating the generated frames to the end
  of the video.

Options:
  -fps, --frame-rate INTEGER      Video FPS. Will default to the input FPS.
  -s, --seconds FLOAT             The total number of seconds to add to the
                                  video. Multiply this number by frame rate to
                                  determine total number of new frames
                                  generated.  [default: 1.0]
  -np, --negative-prompt TEXT     Negative prompt for the diffusion process.
  -cfg, --guidance-scale FLOAT    Guidance scale for the diffusion process.
                                  [default: 7.5]
  -ns, --num-inference-steps INTEGER
                                  Number of diffusion steps.  [default: 50]
  -r, --seed INTEGER              Random seed.
  -m, --model TEXT                HuggingFace model name.
  -nh, --no-half                  Do not use half precision.
  -no, --no-offload               Do not offload to the CPU to preserve GPU
                                  memory.
  -ns, --no-slicing               Do not use VAE slicing.
  -g, --gpu-id INTEGER            GPU ID to use.
  -sf, --model-single-file        Download and use a single file instead of a
                                  directory.
  -cf, --config-file TEXT         Config file to use when using the model-
                                  single-file option. Accepts a path or a
                                  filename in the same directory as the single
                                  file. Will download from the repository
                                  passed in the model option if not provided.
                                  [default: config.json]
  -mf, --model-filename TEXT      The model file to download when using the
                                  model-single-file option.  [default:
                                  vidxtend.safetensors]
  -rs, --remote-subfolder TEXT    Remote subfolder to download from when using
                                  the model-single-file option.
  -cd, --cache-dir DIRECTORY      Cache directory to download to. Default uses
                                  the huggingface cache.
  -o, --output FILE               Output file.  [default: output.mp4]
  -f, --fit [actual|cover|contain|stretch]
                                  Image fit mode.  [default: cover]
  -a, --anchor [top-left|top-center|top-right|center-left|center-center|center-right|bottom-left|bottom-center|bottom-right]
                                  Image anchor point.  [default: top-left]
  --help                          Show this message and exit.

🐍 Python

🔰 Basic Usage

You can create the pipeline, automatically pulling the weights from this repository, either as individual models:

from vidxtend import VidXTendPipeline
pipeline = VidXTendPipeline.from_pretrained(
  "benjamin-paine/vidxtend",
  torch_dtype=torch.float16,
  variant="fp16",
)

Or, as a single file:

from vidxtend import VidXTendPipeline
pipeline = VidXTendPipeline.from_single_file(
  "benjamin-paine/vidxtend",
  torch_dtype=torch.float16,
  variant="fp16",
)

⚙️ Advanced Usage

Use these methods to improve performance:

pipeline.enable_model_cpu_offload()
pipeline.enable_vae_slicing()
pipeline.set_use_memory_efficient_attention_xformers()

Usage is as follows:

# Assume images is a list of PIL Images

new_frames = pipeline(
    prompt=prompt,
    negative_prompt=None, # Optionally use negative prompt
    image=images[-8:], # Use final 8 frames of video
    input_frames_conditioning=images[:1], # Use first frame of video
    eta=1.0,
    guidance_scale=7.5,
    output_type="pil"
).frames[8:] # Remove the first 8 frames from the output as they were used as guide for final 8