Cogvideox Disney Adamw 4000 0.0003 Constant

🚀 CogVideoX LoRA Finetune

This project is a LoRA finetuning of the CogVideoX model, aiming to generate high - quality text - to - video content. It uses advanced training techniques and libraries to optimize the training process and achieve better performance.

✨ Features

• Model Finetuning: Conducts LoRA finetuning on the CogVideoX model THUDM/CogVideoX - 5b.
• Optimized Training: Utilizes CogVideoX Factory with memory - optimized training scripts based on TorchAO and DeepSpeed.
• Easy to Use: Simple usage examples are provided, making it convenient for users to generate videos.

📦 Installation

This project requires the 🧨 Diffusers library to be installed. You can install it using the following command:

pip install diffusers

💻 Usage Examples

Basic Usage

import torch
from diffusers import CogVideoXPipeline
from diffusers.utils import export_to_video

pipe = CogVideoXPipeline.from_pretrained("THUDM/CogVideoX-5b", torch_dtype=torch.bfloat16).to("cuda")
pipe.load_lora_weights("a-r-r-o-w/cogvideox-disney-adamw-4000-0.0003-constant", weight_name="pytorch_lora_weights.safetensors", adapter_name="cogvideox-lora")

# The LoRA adapter weights are determined by what was used for training.
# In this case, we assume `--lora_alpha` is 32 and `--rank` is 64.
# It can be made lower or higher from what was used in training to decrease or amplify the effect
# of the LoRA upto a tolerance, beyond which one might notice no effect at all or overflows.
pipe.set_adapters(["cogvideox-lora"], [32 / 64])

video = pipe("BW_STYLE A black and white animated scene unfolds with an anthropomorphic goat surrounded by musical notes and symbols, suggesting a playful environment. Mickey Mouse appears, leaning forward in curiosity as the goat remains still. The goat then engages with Mickey, who bends down to converse or react. The dynamics shift as Mickey grabs the goat, potentially in surprise or playfulness, amidst a minimalistic background. The scene captures the evolving relationship between the two characters in a whimsical, animated setting, emphasizing their interactions and emotions", guidance_scale=6, use_dynamic_cfg=True).frames[0]
export_to_video(video, "output.mp4", fps=8)

Advanced Usage

For more details, including weighting, merging and fusing LoRAs, check the documentation on loading LoRAs in diffusers.

📚 Documentation

Model description

This is a lora finetune of the CogVideoX model THUDM/CogVideoX - 5b.

The model was trained using CogVideoX Factory - a repository containing memory - optimized training scripts for the CogVideoX family of models using TorchAO and DeepSpeed. The scripts were adopted from CogVideoX Diffusers trainer.

Download model

You can Download LoRA in the Files & Versions tab.

📄 License

Please adhere to the licensing terms as described here and here.

Information Table