Seba-Ai开源视频生成模型 - 支持根据文本描述生成高质量视频

首页

Seba Ai

由 GlitchXRiot 开发

基于CogVideoX-5b的视频生成模型，能够根据文本描述生成高质量视频内容

文本生成视频英语开源协议:MIT #文本生成视频 #高分辨率视频生成 #多语言提示支持

下载量 13

发布时间 : 10/5/2024

模型简介

Seba-AI-5B是一个文本到视频生成模型，源自THUDM的CogVideoX技术，支持从文本提示生成720x480分辨率、6秒时长的视频

模型特点

高质量视频生成

能够根据文本提示生成720x480分辨率、6秒时长的高质量视频

多精度支持

支持BF16、FP16、FP32等多种推理精度，适配不同硬件需求

显存优化

通过diffusers库优化，最低可在4GB显存显卡上运行

量化支持

支持INT8量化，进一步降低显存需求

模型能力

文本生成视频

视频内容创作

创意视觉表达

使用案例

创意内容制作

动画短片创作

根据剧本自动生成动画短片

生成6秒时长的动画视频

广告创意制作

快速生成广告概念视频

生成符合产品描述的广告视频

教育娱乐

教育视频生成

根据教学内容生成可视化视频

生成生动直观的教学视频

游戏场景生成

为游戏开发快速生成场景概念视频

生成符合游戏设定的场景视频

🚀 Seba-AI-5B

Seba-AI-5B 是基于 CogVideoX 的视频生成模型，具有较高的视频生成质量和更好的视觉效果，支持使用 huggingface diffusers 库进行部署。

🚀 快速开始

本模型支持使用 huggingface diffusers 库进行部署，你可以按照以下步骤进行部署。

建议你访问我们的 GitHub 查看相关的提示词优化和转换，以获得更好的体验。

安装所需依赖

# diffusers>=0.30.1
# transformers>=4.44.2
# accelerate>=0.33.0 (建议从源代码安装)
# imageio-ffmpeg>=0.5.1
pip install --upgrade transformers accelerate diffusers imageio-ffmpeg

运行代码

import torch
from diffusers import CogVideoXPipeline
from diffusers.utils import export_to_video

prompt = "A panda, dressed in a small, red jacket and a tiny hat, sits on a wooden stool in a serene bamboo forest. The panda's fluffy paws strum a miniature acoustic guitar, producing soft, melodic tunes. Nearby, a few other pandas gather, watching curiously and some clapping in rhythm. Sunlight filters through the tall bamboo, casting a gentle glow on the scene. The panda's face is expressive, showing concentration and joy as it plays. The background includes a small, flowing stream and vibrant green foliage, enhancing the peaceful and magical atmosphere of this unique musical performance."

pipe = CogVideoXPipeline.from_pretrained(
    "THUDM/CogVideoX-5b",
    torch_dtype=torch.bfloat16
)

pipe.enable_model_cpu_offload()
pipe.vae.enable_tiling()

video = pipe(
    prompt=prompt,
    num_videos_per_prompt=1,
    num_inference_steps=50,
    num_frames=49,
    guidance_scale=6,
    generator=torch.Generator(device="cuda").manual_seed(42),
).frames[0]

export_to_video(video, "output.mp4", fps=8)

✨ 主要特性

高质量视频生成：CogVideoX-5B 模型具有更高的视频生成质量和更好的视觉效果。
多精度支持：支持多种推理精度，如 FP16、BF16、FP32 等。
多 GPU 推理：支持多 GPU 推理，可提高推理速度。
量化支持：可使用 PytorchAO 和 Optimum-quanto 对模型进行量化，降低内存需求。

📦 安装指南

安装所需依赖：

# diffusers>=0.30.1
# transformers>=4.44.2
# accelerate>=0.33.0 (建议从源代码安装)
# imageio-ffmpeg>=0.5.1
pip install --upgrade transformers accelerate diffusers imageio-ffmpeg

💻 使用示例

基础用法

import torch
from diffusers import CogVideoXPipeline
from diffusers.utils import export_to_video

prompt = "A panda, dressed in a small, red jacket and a tiny hat, sits on a wooden stool in a serene bamboo forest. The panda's fluffy paws strum a miniature acoustic guitar, producing soft, melodic tunes. Nearby, a few other pandas gather, watching curiously and some clapping in rhythm. Sunlight filters through the tall bamboo, casting a gentle glow on the scene. The panda's face is expressive, showing concentration and joy as it plays. The background includes a small, flowing stream and vibrant green foliage, enhancing the peaceful and magical atmosphere of this unique musical performance."

pipe = CogVideoXPipeline.from_pretrained(
    "THUDM/CogVideoX-5b",
    torch_dtype=torch.bfloat16
)

pipe.enable_model_cpu_offload()
pipe.vae.enable_tiling()

video = pipe(
    prompt=prompt,
    num_videos_per_prompt=1,
    num_inference_steps=50,
    num_frames=49,
    guidance_scale=6,
    generator=torch.Generator(device="cuda").manual_seed(42),
).frames[0]

export_to_video(video, "output.mp4", fps=8)

高级用法

使用量化推理：

# 开始前，需要从 GitHub 源代码安装 PytorchAO 和 PyTorch Nightly。
# 直到下一个版本发布前，需要进行源代码和夜间版安装。

import torch
from diffusers import AutoencoderKLCogVideoX, CogVideoXTransformer3DModel, CogVideoXPipeline
from diffusers.utils import export_to_video
+ from transformers import T5EncoderModel
+ from torchao.quantization import quantize_, int8_weight_only, int8_dynamic_activation_int8_weight

+ quantization = int8_weight_only

+ text_encoder = T5EncoderModel.from_pretrained("THUDM/CogVideoX-5b", subfolder="text_encoder", torch_dtype=torch.bfloat16)
+ quantize_(text_encoder, quantization())

+ transformer = CogVideoXTransformer3DModel.from_pretrained("THUDM/CogVideoX-5b", subfolder="transformer", torch_dtype=torch.bfloat16)
+ quantize_(transformer, quantization())

+ vae = AutoencoderKLCogVideoX.from_pretrained("THUDM/CogVideoX-5b", subfolder="vae", torch_dtype=torch.bfloat16)
+ quantize_(vae, quantization())

# 创建管道并进行推理
pipe = CogVideoXPipeline.from_pretrained(
    "THUDM/CogVideoX-5b",
+    text_encoder=text_encoder,
+    transformer=transformer,
+    vae=vae,
    torch_dtype=torch.bfloat16,
)
pipe.enable_model_cpu_offload()
pipe.vae.enable_tiling()

prompt = "A panda, dressed in a small, red jacket and a tiny hat, sits on a wooden stool in a serene bamboo forest. The panda's fluffy paws strum a miniature acoustic guitar, producing soft, melodic tunes. Nearby, a few other pandas gather, watching curiously and some clapping in rhythm. Sunlight filters through the tall bamboo, casting a gentle glow on the scene. The panda's face is expressive, showing concentration and joy as it plays. The background includes a small, flowing stream and vibrant green foliage, enhancing the peaceful and magical atmosphere of this unique musical performance."

video = pipe(
    prompt=prompt,
    num_videos_per_prompt=1,
    num_inference_steps=50,
    num_frames=49,
    guidance_scale=6,
    generator=torch.Generator(device="cuda").manual_seed(42),
).frames[0]

export_to_video(video, "output.mp4", fps=8)

📚 详细文档

模型介绍

CogVideoX 是源自 QingYing 的视频生成模型的开源版本。下表展示了我们目前提供的视频生成模型列表及其基础信息。

模型名称	CogVideoX-2B	CogVideoX-5B (本仓库)
模型描述	入门级模型，兼顾兼容性。运行和二次开发成本低。	更大的模型，具有更高的视频生成质量和更好的视觉效果。
推理精度	*FP16 (推荐)*，BF16，FP32，FP8，INT8，不支持 INT4	BF16 (推荐)，FP16，FP32，FP8*，INT8，不支持 INT4
单 GPU 显存消耗	SAT FP16: 18GB diffusers FP16: 从 4GB 起* diffusers INT8(torchao): 从 3.6GB 起*	SAT BF16: 26GB diffusers BF16: 从 5GB 起* diffusers INT8(torchao): 从 4.4GB 起*
多 GPU 推理显存消耗	FP16: 使用 diffusers 为 10GB*	BF16: 使用 diffusers 为 15GB*
推理速度 (Step = 50, FP/BF16)	单 A100: ~90 秒单 H100: ~45 秒	单 A100: ~180 秒单 H100: ~90 秒
微调精度	FP16	BF16
微调显存消耗 (每个 GPU)	47 GB (bs=1, LORA) 61 GB (bs=2, LORA) 62GB (bs=1, SFT)	63 GB (bs=1, LORA) 80 GB (bs=2, LORA) 75GB (bs=1, SFT)
提示词语言	英语*	英语*
提示词长度限制	226 个词元	226 个词元
视频长度	6 秒	6 秒
帧率	每秒 8 帧	每秒 8 帧
视频分辨率	720 x 480，不支持其他分辨率 (包括微调)	720 x 480，不支持其他分辨率 (包括微调)
位置编码	3d_sincos_pos_embed	3d_rope_pos_embed

数据说明

使用 diffusers 库进行测试时，启用了 diffusers 库提供的所有优化。此解决方案尚未在除 NVIDIA A100 / H100 以外的设备上测试实际显存/内存使用情况。一般来说，此解决方案可适用于所有 NVIDIA Ampere 架构 及以上的设备。如果禁用优化，显存使用量将显著增加，峰值显存使用量约为表中显示的 3 倍。不过，速度将提高 3 - 4 倍。你可以选择性地禁用一些优化，包括：

pipe.enable_model_cpu_offload()
pipe.enable_sequential_cpu_offload()
pipe.vae.enable_slicing()
pipe.vae.enable_tiling()

进行多 GPU 推理时，需要禁用 enable_model_cpu_offload() 优化。
使用 INT8 模型会降低推理速度。这是为了确保显存较低的 GPU 能够正常进行推理，同时保持最小的视频质量损失，不过推理速度会显著下降。
2B 模型使用 FP16 精度进行训练，5B 模型使用 BF16 精度进行训练。建议使用模型训练时的精度进行推理。
PytorchAO 和 Optimum-quanto 可用于对文本编码器、Transformer 和 VAE 模块进行量化，以降低 CogVideoX 的内存需求。这使得在免费的 T4 Colab 或显存较小的 GPU 上运行模型成为可能！值得注意的是，TorchAO 量化与 torch.compile 完全兼容，可显著提高推理速度。FP8 精度必须在 NVIDIA H100 及以上的设备上使用，这需要从源代码安装 torch、torchao、diffusers 和 accelerate Python 包。建议使用 CUDA 12.4。
推理速度测试也使用了上述显存优化方案。如果不进行显存优化，推理速度将提高约 10%。只有 diffusers 版本的模型支持量化。
模型仅支持英文输入；其他语言可在细化时由大模型翻译成英文。

注意

使用 SAT 对 SAT 版本的模型进行推理和微调。欢迎访问我们的 GitHub 了解更多信息。

🔧 技术细节

欢迎访问我们的 github，你可以在那里找到：

更详细的技术细节和代码解释。
提示词的优化和转换。
SAT 版本模型的推理和微调，甚至预发布内容。
项目更新日志动态，更多互动机会。
CogVideoX 工具链，帮助你更好地使用模型。
INT8 模型推理代码支持。

📄 许可证

本模型根据 MIT 许可证发布。

引用

@article{GlitchX1.02AbdelrahmanEssa,
  title={Seba-Ai-Video-Generator: Text-to-Video Diffusion Models with An Expert Transformer},
  author={Abdelrahman, Glitch},
  journal={arXiv preprint arXiv:2408.06072},
  year={2024}
}

演示展示

📄 Glitch Readme | 🤗 Glitch-Tech | 🌐 Hugging Fcae |

📍 访问 Glitch H/F 和 API 平台体验商业视频生成模型。

Video Gallery with Captions

A garden comes to life as a kaleidoscope of butterflies flutters amidst the blossoms, their delicate wings casting shadows on the petals below. In the background, a grand fountain cascades water with a gentle splendor, its rhythmic sound providing a soothing backdrop. Beneath the cool shade of a mature tree, a solitary wooden chair invites solitude and reflection, its smooth surface worn by the touch of countless visitors seeking a moment of tranquility in nature's embrace.

A small boy, head bowed and determination etched on his face, sprints through the torrential downpour as lightning crackles and thunder rumbles in the distance. The relentless rain pounds the ground, creating a chaotic dance of water droplets that mirror the dramatic sky's anger. In the far background, the silhouette of a cozy home beckons, a faint beacon of safety and warmth amidst the fierce weather. The scene is one of perseverance and the unyielding spirit of a child braving the elements.

A suited astronaut, with the red dust of Mars clinging to their boots, reaches out to shake hands with an alien being, their skin a shimmering blue, under the pink-tinged sky of the fourth planet. In the background, a sleek silver rocket, a beacon of human ingenuity, stands tall, its engines powered down, as the two representatives of different worlds exchange a historic greeting amidst the desolate beauty of the Martian landscape.

An elderly gentleman, with a serene expression, sits at the water's edge, a steaming cup of tea by his side. He is engrossed in his artwork, brush in hand, as he renders an oil painting on a canvas that's propped up against a small, weathered table. The sea breeze whispers through his silver hair, gently billowing his loose-fitting white shirt, while the salty air adds an intangible element to his masterpiece in progress. The scene is one of tranquility and inspiration, with the artist's canvas capturing the vibrant hues of the setting sun reflecting off the tranquil sea.

In a dimly lit bar, purplish light bathes the face of a mature man, his eyes blinking thoughtfully as he ponders in close-up, the background artfully blurred to focus on his introspective expression, the ambiance of the bar a mere suggestion of shadows and soft lighting.

A golden retriever, sporting sleek black sunglasses, with its lengthy fur flowing in the breeze, sprints playfully across a rooftop terrace, recently refreshed by a light rain. The scene unfolds from a distance, the dog's energetic bounds growing larger as it approaches the camera, its tail wagging with unrestrained joy, while droplets of water glisten on the concrete behind it. The overcast sky provides a dramatic backdrop, emphasizing the vibrant golden coat of the canine as it dashes towards the viewer.

On a brilliant sunny day, the lakeshore is lined with an array of willow trees, their slender branches swaying gently in the soft breeze. The tranquil surface of the lake reflects the clear blue sky, while several elegant swans glide gracefully through the still water, leaving behind delicate ripples that disturb the mirror-like quality of the lake. The scene is one of serene beauty, with the willows' greenery providing a picturesque frame for the peaceful avian visitors.

A Chinese mother, draped in a soft, pastel-colored robe, gently rocks back and forth in a cozy rocking chair positioned in the tranquil setting of a nursery. The dimly lit bedroom is adorned with whimsical mobiles dangling from the ceiling, casting shadows that dance on the walls. Her baby, swaddled in a delicate, patterned blanket, rests against her chest, the child's earlier cries now replaced by contented coos as the mother's soothing voice lulls the little one to sleep. The scent of lavender fills the air, adding to the serene atmosphere, while a warm, orange glow from a nearby nightlight illuminates the scene with a gentle hue, capturing a moment of tender love and comfort.