🚀 首個自動文本到視頻提示補全模型
本模型能夠以幾個單詞作為輸入,生成完整的文本到視頻提示。
🚀 快速開始
本模型基於 Mistral-7B-v0.1,在 VidProM 數據集上進行微調,並使用了 8 塊 A100 GPU。
📦 安裝指南
下載模型
from transformers import pipeline
import torch
pipe = pipeline("text-generation", model="WenhaoWang/AutoT2VPrompt", model_kwargs={"torch_dtype": torch.bfloat16}, device_map="cuda:0")
設置參數
input = "An underwater world"
max_length = 50
temperature = 1.2
top_k = 8
num_return_sequences = 10
生成提示
all_prompts = pipe(input, max_length = max_length, do_sample = True, temperature = temperature, top_k = top_k, num_return_sequences=num_return_sequences)
def process(text):
text = text.replace('\n', '.')
text = text.replace(' .', '.')
text = text[:text.rfind('.')]
text = text + '.'
return text
for i in range(num_return_sequences):
print(process(all_prompts[i]['generated_text']))
運行上述代碼後,你將得到 10 條文本到視頻提示,從中挑選你最喜歡的一條即可。
以下是生成的示例提示:
An underwater world, 25 ye boy, with aqua-green eyes, dk sandy blond hair, from the back, and on his back a fish, 23 ye old, weing glasses,ctoon chacte.
An underwater world, the video should capture the essence of tranquility and the beauty of nature.. a woman with short hair weing a green dress sitting at the desk.
An underwater world, the ocean is full of discded items, the water flows, and the light penetrating through the water.
An underwater world.. a woman with red eyes and red lips is looking forwd.
An underwater world.. an old man sitting in a chair, smoking a pipe, a little smoke coming out of the chair, a man is drinking a glass.
An underwater world. The ocean is filled with bioluminess as the water reflects a soft glow from a bioluminescent phosphorescent light source. The camera slowly moves away and zooms in..
An underwater world. the girl looks at the camera and smiles with happiness..
An underwater world, 1960s horror film..
An underwater world.. 4 men in 1940s style clothes walk ound a gothic castle. night, fe. A girl is running, and there e some flowers along the river.
An underwater world, -camera pan up . A girl is playing with her cat on a sunny day in the pk. A man is running and then falling down and dying.
📄 許可證
本模型採用 CC BY-NC 4.0 許可證。
📚 引用信息
@article{wang2024vidprom,
title={VidProM: A Million-scale Real Prompt-Gallery Dataset for Text-to-Video Diffusion Models},
author={Wang, Wenhao and Yang, Yi},
journal={arXiv preprint arXiv:2403.06098},
year={2024}
}
🙏 致謝
微調過程得到了 Yaowei Zheng 的幫助。
📞 聯繫我們
如果您有任何問題,請隨時聯繫 Wenhao Wang (wangwenhao0716@gmail.com)。