Animatediff Motion Adapter V1 5

🚀 AnimateDiff

AnimateDiff is a method that enables the creation of videos using pre - existing Stable Diffusion Text - to - Image models. It inserts motion module layers into a frozen text - to - image model and trains it on video clips to extract a motion prior.

🚀 Quick Start

AnimateDiff allows you to generate videos from existing Stable Diffusion text - to - image models. It works by integrating motion module layers into a pre - trained text - to - image model and training on video clips to capture motion patterns.

✨ Features

• Motion Module Integration: Inserts motion module layers into a frozen text - to - image model.
• Motion Prior Extraction: Trains on video clips to extract a motion prior.
• Coherent Motion: Introduces coherent motion across image frames.
• MotionAdapter and UNetMotionModel: Concepts introduced to conveniently use motion modules with existing Stable Diffusion models.

💻 Usage Examples

Basic Usage

import torch
from diffusers import MotionAdapter, AnimateDiffPipeline, DDIMScheduler
from diffusers.utils import export_to_gif

# Load the motion adapter
adapter = MotionAdapter.from_pretrained("guoyww/animatediff-motion-adapter-v1-5")
# load SD 1.5 based finetuned model
model_id = "SG161222/Realistic_Vision_V5.1_noVAE"
pipe = AnimateDiffPipeline.from_pretrained(model_id, motion_adapter=adapter)
scheduler = DDIMScheduler.from_pretrained(
    model_id, subfolder="scheduler", clip_sample=False, timestep_spacing="linspace", steps_offset=1
)
pipe.scheduler = scheduler

# enable memory savings
pipe.enable_vae_slicing()
pipe.enable_model_cpu_offload()

output = pipe(
    prompt=(
        "masterpiece, bestquality, highlydetailed, ultradetailed, sunset, "
        "orange sky, warm lighting, fishing boats, ocean waves seagulls, "
        "rippling water, wharf, silhouette, serene atmosphere, dusk, evening glow, "
        "golden hour, coastal landscape, seaside scenery"
    ),
    negative_prompt="bad quality, worse quality",
    num_frames=16,
    guidance_scale=7.5,
    num_inference_steps=25,
    generator=torch.Generator("cpu").manual_seed(42),
)
frames = output.frames[0]
export_to_gif(frames, "animation.gif")

Example Output

Prompt	Output
masterpiece, bestquality, sunset.

⚠️ Important Note

AnimateDiff tends to work better with finetuned Stable Diffusion models. If you plan on using a scheduler that can clip samples, make sure to disable it by setting clip_sample=False in the scheduler as this can also have an adverse effect on generated samples.