๐ LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment
LanguageBind is a language - centric multimodal pretraining approach that uses language as a bind across different modalities. It offers high - performance, a large multimodal dataset, and multi - view enhanced training.
If you like our project, please give us a star โญ on GitHub for latest update.
๐ฐ News
- [2024.01.27] ๐๐๐ Our MoE-LLaVA is released! A sparse model with 3B parameters outperformed the dense model with 7B parameters.
- [2024.01.16] ๐ฅ๐ฅ๐ฅ Our LanguageBind has been accepted at ICLR 2024! We earn the score of 6(3)8(6)6(6)6(6) here.
- [2023.12.15] ๐ช๐ช๐ช We expand the ๐ฅ๐ฅ๐ฅ VIDAL dataset and now have 10M video - text data. We launch LanguageBind_Video 1.5, checking our model zoo.
- [2023.12.10] We expand the ๐ฅ๐ฅ๐ฅ VIDAL dataset and now have 10M depth and 10M thermal data. We are in the process of uploading thermal and depth data on Hugging Face and expect the whole process to last 1 - 2 months.
- [2023.11.27] ๐ฅ๐ฅ๐ฅ We have updated our paper with emergency zero - shot results., checking our โจ results.
- [2023.11.26] ๐ฅ๐ฅ๐ฅ We have open - sourced all textual sources and corresponding YouTube IDs here.
- [2023.11.26] ๐ฃ๐ฃ๐ฃ We have open - sourced fully fine - tuned Video & Audio, achieving improved performance once again, checking our model zoo.
- [2023.11.22] We are about to release a fully fine - tuned version, and the HUGE version is currently undergoing training.
- [2023.11.21] ๐ฅ We are releasing sample data in DATASETS.md so that individuals who are interested can further modify the code to train it on their own data.
- [2023.11.20] ๐๐๐ Video-LLaVA builds a large visual - language model to achieve ๐SOTA performances based on LanguageBind encoders.
- [2023.10.23] ๐ถ LanguageBind - Audio achieves ๐๐๐state - of - the - art (SOTA) performance on 5 datasets, checking our โจ results!
- [2023.10.14] ๐ฑ Released a stronger LanguageBind - Video, checking our โจ results! The video checkpoint have updated on Huggingface Model Hub!
- [2023.10.10] We provide sample data, which can be found in assets, and emergency zero - shot usage is described.
- [2023.10.07] The checkpoints are available on ๐ค Huggingface Model.
- [2023.10.04] Code and demo are available now! Welcome to watch ๐ this repository for the latest updates.
โจ Features
๐ก High performance, but NO intermediate modality required
LanguageBind is a language - centric multimodal pretraining approach, taking the language as the bind across different modalities because the language modality is well - explored and contains rich semantics.
- The following first figure shows the architecture of LanguageBind. LanguageBind can be easily extended to segmentation, detection tasks, and potentially to unlimited modalities.
โก๏ธ A multimodal, fully aligned and voluminous dataset
We propose VIDAL - 10M, 10 Million data with Video, Infrared, Depth, Audio and their corresponding Language, which greatly expands the data beyond visual modalities.
- The second figure shows our proposed VIDAL - 10M dataset, which includes five modalities: video, infrared, depth, audio, and language.
๐ฅ Multi - view enhanced description for training
We make multi - view enhancements to language. We produce multi - view description that combines meta - data, spatial, and temporal to greatly enhance the semantic information of the language. In addition we further enhance the language with ChatGPT to create a good semantic space for each modality aligned language.
๐ค Demo
Basic Usage
python gradio_app.py
Advanced Usage
We provide the online demo in Huggingface Spaces. In this demo, you can calculate the similarity of modalities to language, such as audio - to - language, video - to - language, and depth - to - image.
๐ฆ Installation
- Python >= 3.8
- Pytorch >= 1.13.1
- CUDA Version >= 11.6
- Install required packages:
git clone https://github.com/PKU-YuanGroup/LanguageBind
cd LanguageBind
pip install torch==1.13.1+cu116 torchvision==0.14.1+cu116 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/cu116
pip install -r requirements.txt
๐ณ Model Zoo
The names in the table represent different encoder models. For example, LanguageBind/LanguageBind_Video_FT
represents the fully fine - tuned version, while LanguageBind/LanguageBind_Video
represents the LoRA - tuned version.
You can freely replace them in the recommended API usage. We recommend using the fully fine - tuned version, as it offers stronger performance.
Version |
Tuning |
Model size |
Num_frames |
HF Link |
MSR - VTT |
DiDeMo |
ActivityNet |
MSVD |
LanguageBind_Video |
LoRA |
Large |
8 |
Link |
42.6 |
37.8 |
35.1 |
52.2 |
LanguageBind_Video_FT |
Full - tuning |
Large |
8 |
Link |
42.7 |
38.1 |
36.9 |
53.5 |
LanguageBind_Video_V1.5_FT |
Full - tuning |
Large |
8 |
Link |
42.8 |
39.7 |
38.4 |
54.1 |
LanguageBind_Video_V1.5_FT |
Full - tuning |
Large |
12 |
Coming soon |
- |
- |
- |
- |
LanguageBind_Video_Huge_V1.5_FT |
Full - tuning |
Huge |
8 |
Link |
44.8 |
39.9 |
41.0 |
53.7 |
LanguageBind_Video_Huge_V1.5_FT |
Full - tuning |
Huge |
12 |
Coming soon |
- |
- |
- |
- |
๐ป Usage Examples
Basic Usage
We have provided some sample datasets in assets to quickly see how languagebind works.
import torch
from languagebind import LanguageBind, to_device, transform_dict, LanguageBindImageTokenizer
if __name__ == '__main__':
device = 'cuda:0'
device = torch.device(device)
clip_type = {
'video': 'LanguageBind_Video_FT',
'audio': 'LanguageBind_Audio_FT',
'thermal': 'LanguageBind_Thermal',
'image': 'LanguageBind_Image',
'depth': 'LanguageBind_Depth',
}
model = LanguageBind(clip_type=clip_type, cache_dir='./cache_dir')
model = model.to(device)
model.eval()
pretrained_ckpt = f'lb203/LanguageBind_Image'
tokenizer = LanguageBindImageTokenizer.from_pretrained(pretrained_ckpt, cache_dir='./cache_dir/tokenizer_cache_dir')
modality_transform = {c: transform_dict[c](model.modality_config[c]) for c in clip_type.keys()}
image = ['assets/image/0.jpg', 'assets/image/1.jpg']
audio = ['assets/audio/0.wav', 'assets/audio/1.wav']
video = ['assets/video/0.mp4', 'assets/video/1.mp4']
depth = ['assets/depth/0.png', 'assets/depth/1.png']
thermal = ['assets/thermal/0.jpg', 'assets/thermal/1.jpg']
language = ["Training a parakeet to climb up a ladder.", 'A lion climbing a tree to catch a monkey.']
inputs = {
'image': to_device(modality_transform['image'](image), device),
'video': to_device(modality_transform['video'](video), device),
'audio': to_device(modality_transform['audio'](audio), device),
'depth': to_device(modality_transform['depth'](depth), device),
'thermal': to_device(modality_transform['thermal'](thermal), device),
}
inputs['language'] = to_device(tokenizer(language, max_length=77, padding='max_length',
truncation=True, return_tensors='pt'), device)
with torch.no_grad():
embeddings = model(inputs)
print("Video x Text: \n",
torch.softmax(embeddings['video'] @ embeddings['language'].T, dim=-1).detach().cpu().numpy())
print("Image x Text: \n",
torch.softmax(embeddings['image'] @ embeddings['language'].T, dim=-1).detach().cpu().numpy())
print("Depth x Text: \n",
torch.softmax(embeddings['depth'] @ embeddings['language'].T, dim=-1).detach().cpu().numpy())
print("Audio x Text: \n",
torch.softmax(embeddings['audio'] @ embeddings['language'].T, dim=-1).detach().cpu().numpy())
print("Thermal x Text: \n",
torch.softmax(embeddings['thermal'] @ embeddings['language'].T, dim=-1).detach().cpu().numpy())
Then returns the following result.
Video x Text:
[[9.9989331e-01 1.0667283e-04]
[1.3255903e-03 9.9867439e-01]]
Image x Text:
[[9.9990666e-01 9.3292067e-05]
[4.6132666e-08 1.0000000e+00]]
Depth x Text:
[[0.9954276 0.00457235]
[0.12042473 0.8795753 ]]
Audio x Text:
[[0.97634876 0.02365119]
[0.02917843 0.97082156]]
Thermal x Text:
[[0.9482511 0.0517489 ]
[0.48746133 0.5125386 ]]
Advanced Usage
Emergency zero - shot
Since languagebind binds each modality together, we also found the emergency zero - shot. It's very simple to use.
print("Video x Audio: \n", torch.softmax(embeddings['video'] @ embeddings['audio'].T, dim=-1).detach().cpu().numpy())
print("Image x Depth: \n", torch.softmax(embeddings['image'] @ embeddings['depth'].T, dim=-1).detach().cpu().numpy())
print("Image x Thermal: \n", torch.softmax(embeddings['image'] @ embeddings['thermal'].T, dim=-1).detach().cpu().numpy())
Then, you will get:
Video x Audio:
[[1.0000000e+00 0.0000000e+00]
[3.1150486e-32 1.0000000e+00]]
Image x Depth:
[[1. 0.]
[0. 1.]]
Image x Thermal:
[[1. 0.]
[0. 1.]]
Different branches for X - Language task
Additionally, LanguageBind can be disassembled into different branches to handle different tasks. Note that we do not train Image, which just initialize from OpenCLIP.
Thermal
import torch
from languagebind import LanguageBindThermal, LanguageBindThermalTokenizer, LanguageBindThermalProcessor
pretrained_ckpt = 'LanguageBind/LanguageBind_Thermal'
model = LanguageBindThermal.from_pretrained(pretrained_ckpt, cache_dir='./cache_dir')
tokenizer = LanguageBindThermalTokenizer.from_pretrained(pretrained_ckpt, cache_dir='./cache_dir')
thermal_process = LanguageBindThermalProcessor(model.config, tokenizer)
model.eval()
data = thermal_process([r"your/thermal.jpg"], ['your text'], return_tensors='pt')
with torch.no_grad():
out = model(**data)
print(out.text_embeds @ out.image_embeds.T)
Depth
import torch
from languagebind import ...
๐ License
This project is licensed under the MIT license.