Animatediff Motion Adapter V1 4

🚀 AnimateDiff: A Text-to-Video Solution with Diffusers

AnimateDiff is a method that enables the creation of videos using pre - existing Stable Diffusion Text - to - Image models. It inserts motion module layers into a frozen text - to - image model and trains it on video clips to extract a motion prior.

🚀 Quick Start

AnimateDiff is a method that allows you to create videos using pre - existing Stable Diffusion Text to Image models. It inserts motion module layers into a frozen text - to - image model and trains it on video clips to extract a motion prior. These motion modules are applied after the ResNet and Attention blocks in the Stable Diffusion UNet to introduce coherent motion across image frames. To support these modules, the concepts of a MotionAdapter and UNetMotionModel are introduced, which serve as a convenient way to use these motion modules with existing Stable Diffusion models.

✨ Features

• Insert motion module layers into a frozen text - to - image model to extract motion prior.
• Introduce coherent motion across image frames by applying motion modules after ResNet and Attention blocks in the Stable Diffusion UNet.
• Use MotionAdapter and UNetMotionModel to conveniently integrate motion modules with existing Stable Diffusion models.

💻 Usage Examples

Basic Usage

The following example demonstrates how you can utilize the motion modules with an existing Stable Diffusion text to image model.

import torch
from diffusers import MotionAdapter, AnimateDiffPipeline, DDIMScheduler
from diffusers.utils import export_to_gif

# Load the motion adapter
adapter = MotionAdapter.from_pretrained("guoyww/animatediff-motion-adapter-v1-4")
model_id = "SG161222/Realistic_Vision_V5.1_noVAE"
pipe = AnimateDiffPipeline.from_pretrained(model_id, motion_adapter=adapter)
scheduler = DDIMScheduler.from_pretrained(
    model_id, subfolder="scheduler", clip_sample=False, timestep_spacing="linspace", steps_offset=1
)
pipe.scheduler = scheduler

# enable memory savings
pipe.enable_vae_slicing()
pipe.enable_model_cpu_offload()

output = pipe(
    prompt=(
        "masterpiece, bestquality, highlydetailed, ultradetailed, sunset, "
        "orange sky, warm lighting, fishing boats, ocean waves seagulls, "
        "rippling water, wharf, silhouette, serene atmosphere, dusk, evening glow, "
        "golden hour, coastal landscape, seaside scenery"
    ),
    negative_prompt="bad quality, worse quality",
    num_frames=16,
    guidance_scale=7.5,
    num_inference_steps=25,
    generator=torch.Generator("cpu").manual_seed(42),
)
frames = output.frames[0]
export_to_gif(frames, "animation.gif")

Example Output

masterpiece, bestquality, sunset.

⚠️ Important Note

AnimateDiff tends to work better with finetuned Stable Diffusion models. If you plan on using a scheduler that can clip samples, make sure to disable it by setting clip_sample=False in the scheduler as this can also have an adverse effect on generated samples.