Languagebind Image

🚀 LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment

LanguageBind is a novel multimodal pretraining approach that extends video - language pretraining to N - modality through language - based semantic alignment. It offers high - performance solutions with a large - scale multimodal dataset and innovative training methods.

📦 Installation

Requirements

• Python >= 3.8
• Pytorch >= 1.13.1
• CUDA Version >= 11.6

Steps

git clone https://github.com/PKU-YuanGroup/LanguageBind
cd LanguageBind
pip install torch==1.13.1+cu116 torchvision==0.14.1+cu116 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/cu116
pip install -r requirements.txt

✨ Features

💡 High performance, but NO intermediate modality required

LanguageBind is a language - centric multimodal pretraining approach. It uses language as the bind across different modalities because the language modality is well - explored and contains rich semantics. The architecture of LanguageBind can be easily extended to segmentation, detection tasks, and potentially to unlimited modalities.

⚡️ A multimodal, fully aligned and voluminous dataset

We propose VIDAL - 10M, a dataset with 10 million data points including Video, Infrared, Depth, Audio and their corresponding Language, which greatly expands the data beyond visual modalities.

🔥 Multi - view enhanced description for training

We make multi - view enhancements to language. We produce multi - view descriptions that combine meta - data, spatial, and temporal information to greatly enhance the semantic information of the language. In addition, we further enhance the language with ChatGPT to create a good semantic space for each modality - aligned language.

📰 News

• [2024.01.27] 👀👀👀 Our [MoE - LLaVA](https://github.com/PKU - YuanGroup/MoE - LLaVA) is released! A sparse model with 3B parameters outperformed the dense model with 7B parameters.
• [2024.01.16] 🔥🔥🔥 Our LanguageBind has been accepted at ICLR 2024! We earn the score of 6(3)8(6)6(6)6(6) here.
• [2023.12.15] 💪💪💪 We expand the 💥💥💥 VIDAL dataset and now have 10M video - text data. We launch LanguageBind_Video 1.5, check our model zoo.
• [2023.12.10] We expand the 💥💥💥 VIDAL dataset and now have 10M depth and 10M thermal data. We are in the process of uploading thermal and depth data on [Hugging Face](https://huggingface.co/datasets/LanguageBind/VIDAL - Depth - Thermal) and expect the whole process to last 1 - 2 months.
• [2023.11.27] 🔥🔥🔥 We have updated our paper with emergency zero - shot results. Check our ✨ [results](#multiple - modalities)!
• [2023.11.26] 💥💥💥 We have open - sourced all textual sources and corresponding YouTube IDs here.
• [2023.11.26] 📣📣📣 We have open - sourced fully fine - tuned Video & Audio, achieving improved performance once again. Check our model zoo.
• [2023.11.22] We are about to release a fully fine - tuned version, and the HUGE version is currently undergoing training.
• [2023.11.21] 💥 We are releasing sample data in DATASETS.md so that individuals who are interested can further modify the code to train it on their own data.
• [2023.11.20] 🚀🚀🚀 [Video - LLaVA](https://github.com/PKU - YuanGroup/Video - LLaVA) builds a large visual - language model to achieve 🎉SOTA performances based on LanguageBind encoders.
• [2023.10.23] 🎶 LanguageBind - Audio achieves 🎉🎉🎉state - of - the - art (SOTA) performance on 5 datasets. Check our ✨ [results](#multiple - modalities)!
• [2023.10.14] 😱 Released a stronger LanguageBind - Video. Check our ✨ [results](#video - language)! The video checkpoint has updated on Huggingface Model Hub!
• [2023.10.10] We provide sample data, which can be found in assets, and [emergency zero - shot usage](#emergency - zero - shot) is described.
• [2023.10.07] The checkpoints are available on 🤗 Huggingface Model.
• [2023.10.04] Code and demo are available now! Welcome to watch 👀 this repository for the latest updates.

🤗 Demo

Local demo

Highly recommend trying out our web demo, which incorporates all features currently supported by LanguageBind.

python gradio_app.py

Online demo

We provide the online demo in Huggingface Spaces. In this demo, you can calculate the similarity of modalities to language, such as audio - to - language, video - to - language, and depth - to - image.

🐳 Model Zoo

The names in the table represent different encoder models. For example, LanguageBind/LanguageBind_Video_FT represents the fully fine - tuned version, while LanguageBind/LanguageBind_Video represents the LoRA - tuned version. You can freely replace them in the recommended API usage. We recommend using the fully fine - tuned version, as it offers stronger performance.

💻 Usage Examples

Basic Usage

We have provided some sample datasets in assets to quickly see how languagebind works.

import torch
from languagebind import LanguageBind, to_device, transform_dict, LanguageBindImageTokenizer

if __name__ == '__main__':
    device = 'cuda:0'
    device = torch.device(device)
    clip_type = {
        'video': 'LanguageBind_Video_FT',  # also LanguageBind_Video
        'audio': 'LanguageBind_Audio_FT',  # also LanguageBind_Audio
        'thermal': 'LanguageBind_Thermal',
        'image': 'LanguageBind_Image',
        'depth': 'LanguageBind_Depth',
    }

    model = LanguageBind(clip_type=clip_type, cache_dir='./cache_dir')
    model = model.to(device)
    model.eval()
    pretrained_ckpt = f'lb203/LanguageBind_Image'
    tokenizer = LanguageBindImageTokenizer.from_pretrained(pretrained_ckpt, cache_dir='./cache_dir/tokenizer_cache_dir')
    modality_transform = {c: transform_dict[c](model.modality_config[c]) for c in clip_type.keys()}

    image = ['assets/image/0.jpg', 'assets/image/1.jpg']
    audio = ['assets/audio/0.wav', 'assets/audio/1.wav']
    video = ['assets/video/0.mp4', 'assets/video/1.mp4']
    depth = ['assets/depth/0.png', 'assets/depth/1.png']
    thermal = ['assets/thermal/0.jpg', 'assets/thermal/1.jpg']
    language = ["Training a parakeet to climb up a ladder.", 'A lion climbing a tree to catch a monkey.']

    inputs = {
        'image': to_device(modality_transform['image'](image), device),
        'video': to_device(modality_transform['video'](video), device),
        'audio': to_device(modality_transform['audio'](audio), device),
        'depth': to_device(modality_transform['depth'](depth), device),
        'thermal': to_device(modality_transform['thermal'](thermal), device),
    }
    inputs['language'] = to_device(tokenizer(language, max_length=77, padding='max_length',
                                             truncation=True, return_tensors='pt'), device)

    with torch.no_grad():
        embeddings = model(inputs)

    print("Video x Text: \n",
          torch.softmax(embeddings['video'] @ embeddings['language'].T, dim=-1).detach().cpu().numpy())
    print("Image x Text: \n",
          torch.softmax(embeddings['image'] @ embeddings['language'].T, dim=-1).detach().cpu().numpy())
    print("Depth x Text: \n",
          torch.softmax(embeddings['depth'] @ embeddings['language'].T, dim=-1).detach().cpu().numpy())
    print("Audio x Text: \n",
          torch.softmax(embeddings['audio'] @ embeddings['language'].T, dim=-1).detach().cpu().numpy())
    print("Thermal x Text: \n",
          torch.softmax(embeddings['thermal'] @ embeddings['language'].T, dim=-1).detach().cpu().numpy())

The output will be:

Video x Text: 
 [[9.9989331e-01 1.0667283e-04]
 [1.3255903e-03 9.9867439e-01]]
Image x Text: 
 [[9.9990666e-01 9.3292067e-05]
 [4.6132666e-08 1.0000000e+00]]
Depth x Text: 
 [[0.9954276  0.00457235]
 [0.12042473 0.8795753 ]]
Audio x Text: 
 [[0.97634876 0.02365119]
 [0.02917843 0.97082156]]
Thermal x Text: 
 [[0.9482511  0.0517489 ]
 [0.48746133 0.5125386 ]]

Advanced Usage

Emergency zero - shot

Since languagebind binds each modality together, we also found the emergency zero - shot. It's very simple to use.

print("Video x Audio: \n", torch.softmax(embeddings['video'] @ embeddings['audio'].T, dim=-1).detach().cpu().numpy())
print("Image x Depth: \n", torch.softmax(embeddings['image'] @ embeddings['depth'].T, dim=-1).detach().cpu().numpy())
print("Image x Thermal: \n", torch.softmax(embeddings['image'] @ embeddings['thermal'].T, dim=-1).detach().cpu().numpy())

The output will be:

Video x Audio: 
 [[1.0000000e+00 0.0000000e+00]
 [3.1150486e-32 1.0000000e+00]]
Image x Depth: 
 [[1. 0.]
 [0. 1.]]
Image x Thermal: 
 [[1. 0.]
 [0. 1.]]

Different branches for X - Language task

Additionally, LanguageBind can be disassembled into different branches to handle different tasks. Note that we do not train Image, which just initialize from OpenCLIP.

Thermal

import torch
from languagebind import LanguageBindThermal, LanguageBindThermalTokenizer, LanguageBindThermalProcessor

pretrained_ckpt = 'LanguageBind/LanguageBind_Thermal'
model = LanguageBindThermal.from_pretrained(pretrained_ckpt, cache_dir='./cache_dir')
tokenizer = LanguageBindThermalTokenizer.from_pretrained(pretrained_ckpt, cache_dir='./cache_dir')
thermal_process = LanguageBindThermalProcessor(model.config, tokenizer)

model.eval()
data = thermal_process([r"your/thermal.jpg"], ['your text'], return_tensors='pt')
with torch.no_grad():
    out = model(**data)

print(out.text_embeds @ out.image_embeds.T)

📄 License

This project is licensed under the MIT license.