Altclip

🚀 AltCLIP

A simple and efficient method to train a better bilingual CLIP model

🚀 Quick Start

AltCLIP is a bilingual CLIP model that can provide support for the AltDiffusion model. You can download the code from FlagAI AltCLIP and follow the instructions in the README to get started.

✨ Features

• Bilingual Support: AltCLIP supports both Chinese and English, making it suitable for a wide range of applications.
• High Performance: The model shows excellent performance in text-image retrieval tasks, outperforming many existing models.
• Open Source: The model code is open source on FlagAI, and the weights are available on modelhub.

📋 Model Information

📚 Documentation

Brief Introduction

We propose a simple and efficient method to train a better bilingual CLIP model, named AltCLIP. AltCLIP is trained based on Stable Diffusiosn, and the training data comes from WuDao dataset and LIAON.

The AltCLIP model can support the AltDiffusion model in this project. For more information about the AltDiffusion model, please refer to this tutorial.

Training

There are two phases of training. In the parallel knowledge distillation phase, we only use parallel corpus texts for distillation (parallel corpus is easier to obtain and larger in number compared to image text pairs). In the bilingual comparison learning phase, we use a small number of Chinese-English image-text pairs (about 2 million in total) to train our text encoder to better fit the image encoder.

Performance

The following table shows the performance of AltCLIP and other models in text-to-image and image-to-text retrieval tasks:

Visualization effects

Based on AltCLIP, we have also developed the AltDiffusion model, visualized as follows:

💻 Usage Examples

Basic Usage

from PIL import Image
import requests

# transformers version >= 4.21.0
from modeling_altclip import AltCLIP
from processing_altclip import AltCLIPProcessor

# now our repo's in private, so we need `use_auth_token=True`
model = AltCLIP.from_pretrained("BAAI/AltCLIP")
processor = AltCLIPProcessor.from_pretrained("BAAI/AltCLIP")

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

inputs = processor(text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True)

outputs = model(**inputs)
logits_per_image = outputs.logits_per_image # this is the image-text similarity score
probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities

🔧 Technical Details

Training Phases

The training of AltCLIP consists of two phases:

1. Parallel Knowledge Distillation Phase: We use parallel corpus texts for distillation. Parallel corpus is easier to obtain and larger in quantity compared to image-text pairs.
2. Bilingual Contrastive Learning Phase: We use a small number (about 2 million) of Chinese-English image-text pairs to train the text encoder to better match the image encoder.

📄 License

This model is licensed under the CreativeML OpenRAIL-M license. By accessing the repository, you agree to the terms of the license and that your contact information (email address and username) may be shared with the model authors.

📚 Citation

If you find this work helpful, please consider citing the following paper:

@article{https://doi.org/10.48550/arxiv.2211.06679,
  doi = {10.48550/ARXIV.2211.06679},
  url = {https://arxiv.org/abs/2211.06679},
  author = {Chen, Zhongzhi and Liu, Guang and Zhang, Bo-Wen and Ye, Fulong and Yang, Qinghong and Wu, Ledell},
  keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences},
  title = {AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilities},
  publisher = {arXiv},
  year = {2022},
  copyright = {arXiv.org perpetual, non-exclusive license}
}