Ola 7b
Ola-7B是由騰訊、清華大學和南洋理工大學聯合開發的多模態大語言模型,基於Qwen2.5架構,支持處理文本、圖像、視頻和音頻輸入,並生成文本輸出。
下載量 1,020
發布時間 : 1/25/2025
模型概述
Ola-7B是一個多模態大語言模型,能夠同時處理圖像/視頻、文本和音頻輸入,並輸出文本。它提供了一種按需解決方案,能夠無縫且高效地處理任意空間尺寸和時間長度的視覺輸入。
模型特點
多模態處理能力
支持同時處理文本、圖像、視頻和音頻輸入,實現跨模態理解與交互。
大上下文窗口
支持32K tokens的上下文窗口,適合處理長文本和多輪對話。
高效視覺處理
能夠無縫且高效地處理任意空間尺寸和時間長度的視覺輸入。
模型能力
文本理解與生成
圖像理解
視頻理解
語音理解
多模態交互
使用案例
智能助手
多模態對話
通過結合圖像、視頻和語音輸入,提供更豐富的對話體驗。
內容理解
視頻內容分析
分析視頻內容並生成描述性文本。
🚀 Ola-7B
Ola-7B模型由來自騰訊、清華大學和南洋理工大學的人員開發。它基於Qwen2.5語言模型,在文本、圖像、視頻和音頻數據上進行訓練,上下文窗口為32K個標記。該模型可以接受圖像、視頻、文本和音頻作為輸入,並輸出文本。Ola提供了一種按需解決方案,能夠無縫高效地處理任意空間大小和時間長度的視覺輸入。
🚀 快速開始
準備工作
- 從https://huggingface.co/THUdyh/Ola_speech_encoders 下載語音編碼器。
- 將
config.json
中的路徑替換為語音編碼器的本地路徑。
簡單使用示例
我們提供了一個簡單的生成過程來使用我們的模型。更多詳細信息,請參考我們的Github倉庫。
import os
os.environ['LOWRES_RESIZE'] = '384x32'
os.environ['HIGHRES_BASE'] = '0x32'
os.environ['VIDEO_RESIZE'] = "0x64"
os.environ['VIDEO_MAXRES'] = "480"
os.environ['VIDEO_MINRES'] = "288"
os.environ['MAXRES'] = '1536'
os.environ['MINRES'] = '0'
os.environ['REGIONAL_POOL'] = '2x'
os.environ['FORCE_NO_DOWNSAMPLE'] = '1'
os.environ['LOAD_VISION_EARLY'] = '1'
os.environ['SKIP_LOAD_VIT'] = '1'
import gradio as gr
import torch
import re
from decord import VideoReader, cpu
from PIL import Image
import numpy as np
import transformers
import moviepy.editor as mp
from typing import Dict, Optional, Sequence, List
import librosa
import whisper
from ola.conversation import conv_templates, SeparatorStyle
from ola.model.builder import load_pretrained_model
from ola.utils import disable_torch_init
from ola.datasets.preprocess import tokenizer_image_token, tokenizer_speech_image_token, tokenizer_speech_question_image_token
from ola.mm_utils import get_model_name_from_path, KeywordsStoppingCriteria, process_anyres_video, process_anyres_highres_image_genli
from ola.constants import IGNORE_INDEX, DEFAULT_IMAGE_TOKEN, IMAGE_TOKEN_INDEX, DEFAULT_SPEECH_TOKEN
model_path = ""
tokenizer, model, image_processor, _ = load_pretrained_model(model_path, None)
model = model.to('cuda').eval()
model = model.bfloat16()
USE_SPEECH=False
cur_dir = os.path.dirname(os.path.abspath(__file__))
def load_audio(audio_file_name):
speech_wav, samplerate = librosa.load(audio_file_name, sr=16000)
if len(speech_wav.shape) > 1:
speech_wav = speech_wav[:, 0]
speech_wav = speech_wav.astype(np.float32)
CHUNK_LIM = 480000
SAMPLE_RATE = 16000
speechs = []
speech_wavs = []
if len(speech_wav) <= CHUNK_LIM:
speech = whisper.pad_or_trim(speech_wav)
speech_wav = whisper.pad_or_trim(speech_wav)
speechs.append(speech)
speech_wavs.append(torch.from_numpy(speech_wav).unsqueeze(0))
else:
for i in range(0, len(speech_wav), CHUNK_LIM):
chunk = speech_wav[i : i + CHUNK_LIM]
if len(chunk) < CHUNK_LIM:
chunk = whisper.pad_or_trim(chunk)
speechs.append(chunk)
speech_wavs.append(torch.from_numpy(chunk).unsqueeze(0))
mels = []
for chunk in speechs:
chunk = whisper.log_mel_spectrogram(chunk, n_mels=128).permute(1, 0).unsqueeze(0)
mels.append(chunk)
mels = torch.cat(mels, dim=0)
speech_wavs = torch.cat(speech_wavs, dim=0)
if mels.shape[0] > 25:
mels = mels[:25]
speech_wavs = speech_wavs[:25]
speech_length = torch.LongTensor([mels.shape[1]] * mels.shape[0])
speech_chunks = torch.LongTensor([mels.shape[0]])
return mels, speech_length, speech_chunks, speech_wavs
def extract_audio(videos_file_path):
my_clip = mp.VideoFileClip(videos_file_path)
return my_clip.audio
def ola_inference(multimodal, audio_path):
visual, text = multimodal["files"][0], multimodal["text"]
if visual.endswith("image2.png"):
modality = "video"
visual = f"{cur_dir}/case/case1.mp4"
if visual.endswith(".mp4"):
modality = "video"
else:
modality = "image"
# input audio and video, do not parse audio in the video, else parse audio in the video
if audio_path:
USE_SPEECH = True
elif modality == "video":
USE_SPEECH = True
else:
USE_SPEECH = False
speechs = []
speech_lengths = []
speech_wavs = []
speech_chunks = []
if modality == "video":
vr = VideoReader(visual, ctx=cpu(0))
total_frame_num = len(vr)
fps = round(vr.get_avg_fps())
uniform_sampled_frames = np.linspace(0, total_frame_num - 1, 64, dtype=int)
frame_idx = uniform_sampled_frames.tolist()
spare_frames = vr.get_batch(frame_idx).asnumpy()
video = [Image.fromarray(frame) for frame in spare_frames]
else:
image = [Image.open(visual)]
image_sizes = [image[0].size]
if USE_SPEECH and audio_path:
audio_path = audio_path
speech, speech_length, speech_chunk, speech_wav = load_audio(audio_path)
speechs.append(speech.bfloat16().to('cuda'))
speech_lengths.append(speech_length.to('cuda'))
speech_chunks.append(speech_chunk.to('cuda'))
speech_wavs.append(speech_wav.to('cuda'))
print('load audio')
elif USE_SPEECH and not audio_path:
# parse audio in the video
audio = extract_audio(visual)
audio.write_audiofile("./video_audio.wav")
video_audio_path = './video_audio.wav'
speech, speech_length, speech_chunk, speech_wav = load_audio(video_audio_path)
speechs.append(speech.bfloat16().to('cuda'))
speech_lengths.append(speech_length.to('cuda'))
speech_chunks.append(speech_chunk.to('cuda'))
speech_wavs.append(speech_wav.to('cuda'))
else:
speechs = [torch.zeros(1, 3000, 128).bfloat16().to('cuda')]
speech_lengths = [torch.LongTensor([3000]).to('cuda')]
speech_wavs = [torch.zeros([1, 480000]).to('cuda')]
speech_chunks = [torch.LongTensor([1]).to('cuda')]
conv_mode = "qwen_1_5"
if text:
qs = text
else:
qs = ''
if USE_SPEECH and audio_path:
qs = DEFAULT_IMAGE_TOKEN + "\n" + "User's question in speech: " + DEFAULT_SPEECH_TOKEN + '\n'
elif USE_SPEECH:
qs = DEFAULT_SPEECH_TOKEN + DEFAULT_IMAGE_TOKEN + "\n" + qs
else:
qs = DEFAULT_IMAGE_TOKEN + "\n" + qs
conv = conv_templates[conv_mode].copy()
conv.append_message(conv.roles[0], qs)
conv.append_message(conv.roles[1], None)
prompt = conv.get_prompt()
if USE_SPEECH and audio_path:
input_ids = tokenizer_speech_question_image_token(prompt, tokenizer, IMAGE_TOKEN_INDEX, return_tensors="pt").unsqueeze(0).to('cuda')
elif USE_SPEECH:
input_ids = tokenizer_speech_image_token(prompt, tokenizer, IMAGE_TOKEN_INDEX, return_tensors="pt").unsqueeze(0).to('cuda')
else:
input_ids = tokenizer_image_token(prompt, tokenizer, IMAGE_TOKEN_INDEX, return_tensors="pt").unsqueeze(0).to('cuda')
if modality == "video":
video_processed = []
for idx, frame in enumerate(video):
image_processor.do_resize = False
image_processor.do_center_crop = False
frame = process_anyres_video(frame, image_processor)
if frame_idx is not None and idx in frame_idx:
video_processed.append(frame.unsqueeze(0))
elif frame_idx is None:
video_processed.append(frame.unsqueeze(0))
if frame_idx is None:
frame_idx = np.arange(0, len(video_processed), dtype=int).tolist()
video_processed = torch.cat(video_processed, dim=0).bfloat16().to("cuda")
video_processed = (video_processed, video_processed)
video_data = (video_processed, (384, 384), "video")
else:
image_processor.do_resize = False
image_processor.do_center_crop = False
image_tensor, image_highres_tensor = [], []
for visual in image:
image_tensor_, image_highres_tensor_ = process_anyres_highres_image_genli(visual, image_processor)
image_tensor.append(image_tensor_)
image_highres_tensor.append(image_highres_tensor_)
if all(x.shape == image_tensor[0].shape for x in image_tensor):
image_tensor = torch.stack(image_tensor, dim=0)
if all(x.shape == image_highres_tensor[0].shape for x in image_highres_tensor):
image_highres_tensor = torch.stack(image_highres_tensor, dim=0)
if type(image_tensor) is list:
image_tensor = [_image.bfloat16().to("cuda") for _image in image_tensor]
else:
image_tensor = image_tensor.bfloat16().to("cuda")
if type(image_highres_tensor) is list:
image_highres_tensor = [_image.bfloat16().to("cuda") for _image in image_highres_tensor]
else:
image_highres_tensor = image_highres_tensor.bfloat16().to("cuda")
pad_token_ids = 151643
attention_masks = input_ids.ne(pad_token_ids).long().to('cuda')
stop_str = conv.sep if conv.sep_style != SeparatorStyle.TWO else conv.sep2
keywords = [stop_str]
stopping_criteria = KeywordsStoppingCriteria(keywords, tokenizer, input_ids)
gen_kwargs = {}
if "max_new_tokens" not in gen_kwargs:
gen_kwargs["max_new_tokens"] = 1024
if "temperature" not in gen_kwargs:
gen_kwargs["temperature"] = 0.2
if "top_p" not in gen_kwargs:
gen_kwargs["top_p"] = None
if "num_beams" not in gen_kwargs:
gen_kwargs["num_beams"] = 1
with torch.inference_mode():
if modality == "video":
output_ids = model.generate(
inputs=input_ids,
images=video_data[0][0],
images_highres=video_data[0][1],
modalities=video_data[2],
speech=speechs,
speech_lengths=speech_lengths,
speech_chunks=speech_chunks,
speech_wav=speech_wavs,
attention_mask=attention_masks,
use_cache=True,
stopping_criteria=[stopping_criteria],
do_sample=True if gen_kwargs["temperature"] > 0 else False,
temperature=gen_kwargs["temperature"],
top_p=gen_kwargs["top_p"],
num_beams=gen_kwargs["num_beams"],
max_new_tokens=gen_kwargs["max_new_tokens"],
)
else:
output_ids = model.generate(
inputs=input_ids,
images=image_tensor,
images_highres=image_highres_tensor,
image_sizes=image_sizes,
modalities=['image'],
speech=speechs,
speech_lengths=speech_lengths,
speech_chunks=speech_chunks,
speech_wav=speech_wavs,
attention_mask=attention_masks,
use_cache=True,
stopping_criteria=[stopping_criteria],
do_sample=True if gen_kwargs["temperature"] > 0 else False,
temperature=gen_kwargs["temperature"],
top_p=gen_kwargs["top_p"],
num_beams=gen_kwargs["num_beams"],
max_new_tokens=gen_kwargs["max_new_tokens"],
)
outputs = tokenizer.batch_decode(output_ids, skip_special_tokens=True)[0]
outputs = outputs.strip()
if outputs.endswith(stop_str):
outputs = outputs[:-len(stop_str)]
outputs = outputs.strip()
return outputs, None
✨ 主要特性
- 支持多模態輸入:可以接受圖像、視頻、文本和音頻作為輸入,並輸出文本。
- 按需處理視覺輸入:提供按需解決方案,能夠無縫高效地處理任意空間大小和時間長度的視覺輸入。
📚 詳細文檔
模型架構
- 架構:預訓練的Oryx-ViT + Qwen2.5-7B
- 數據:超過500萬的圖像、視頻和音頻數據的混合,分3個階段進行訓練。
- 精度:BFloat16
硬件與軟件
- 硬件:64 * NVIDIA Tesla A100
- 編排:HuggingFace Trainer
- 代碼:Pytorch
📄 許可證
本項目採用Apache-2.0許可證。
📚 引用
@article{liu2025ola,
title={Ola: Pushing the Frontiers of Omni-Modal Language Model with Progressive Modality Alignment},
author={Liu, Zuyan and Dong, Yuhao and Wang, Jiahui and Liu, Ziwei and Hu, Winston and Lu, Jiwen and Rao, Yongming},
journal={arXiv preprint arXiv:2502.04328},
year={2025}
}
Codebert Base
CodeBERT是一個面向編程語言與自然語言的預訓練模型,基於RoBERTa架構,支持代碼搜索和代碼生成文檔等功能。
多模態融合
C
microsoft
1.6M
248
Llama 4 Scout 17B 16E Instruct
其他
Llama 4 Scout是Meta開發的多模態AI模型,採用混合專家架構,支持12種語言的文本和圖像交互,具有17B激活參數和109B總參數。
多模態融合
Transformers 支持多種語言

L
meta-llama
817.62k
844
Unixcoder Base
Apache-2.0
UniXcoder是一個統一的多模態預訓練模型,利用代碼註釋和抽象語法樹等多模態數據預訓練代碼表示。
多模態融合
Transformers 英語

U
microsoft
347.45k
51
TITAN
TITAN是一個多模態全切片基礎模型,通過視覺自監督學習和視覺-語言對齊進行預訓練,用於病理學圖像分析。
多模態融合
Safetensors 英語
T
MahmoodLab
213.39k
37
Qwen2.5 Omni 7B
其他
Qwen2.5-Omni 是一個端到端的多模態模型,能夠感知文本、圖像、音頻和視頻等多種模態,並以流式方式生成文本和自然語音響應。
多模態融合
Transformers 英語

Q
Qwen
206.20k
1,522
Minicpm O 2 6
MiniCPM-o 2.6是一款手機端運行的GPT-4o級多模態大模型,支持視覺、語音與直播流處理
多模態融合
Transformers 其他

M
openbmb
178.38k
1,117
Llama 4 Scout 17B 16E Instruct
其他
Llama 4 Scout是Meta推出的17B參數/16專家混合的多模態AI模型,支持12種語言和圖像理解,具有行業領先性能。
多模態融合
Transformers 支持多種語言

L
chutesai
173.52k
2
Qwen2.5 Omni 3B
其他
Qwen2.5-Omni是一款端到端多模態模型,能夠感知文本、圖像、音頻和視頻等多種模態信息,並以流式方式同步生成文本和自然語音響應。
多模態融合
Transformers 英語

Q
Qwen
48.07k
219
One Align
MIT
Q-Align是一個多任務視覺評估模型,專注於圖像質量評估(IQA)、美學評估(IAA)和視頻質量評估(VQA),在ICML2024上發表。
多模態融合
Transformers

O
q-future
39.48k
25
Biomedvlp BioViL T
MIT
BioViL-T是一個專注於分析胸部X光片和放射學報告的視覺語言模型,通過時序多模態預訓練提升性能。
多模態融合
Transformers 英語

B
microsoft
26.39k
35
精選推薦AI模型
Llama 3 Typhoon V1.5x 8b Instruct
專為泰語設計的80億參數指令模型,性能媲美GPT-3.5-turbo,優化了應用場景、檢索增強生成、受限生成和推理任務
大型語言模型
Transformers 支持多種語言

L
scb10x
3,269
16
Cadet Tiny
Openrail
Cadet-Tiny是一個基於SODA數據集訓練的超小型對話模型,專為邊緣設備推理設計,體積僅為Cosmo-3B模型的2%左右。
對話系統
Transformers 英語

C
ToddGoldfarb
2,691
6
Roberta Base Chinese Extractive Qa
基於RoBERTa架構的中文抽取式問答模型,適用於從給定文本中提取答案的任務。
問答系統 中文
R
uer
2,694
98