Nllb Clip Base Siglip

🚀 NLLB-CLIP-SigLIP Model

NLLB-CLIP-SigLIP combines a text encoder from the NLLB model and an image encoder from the SigLIP model. It extends model capabilities to 201 languages of the Flores - 200 and sets state - of - the - art on the Crossmodal - 3600 dataset, especially excelling in low - resource languages.

📦 Model Information

✨ Features

• Combines text encoder from NLLB model and image encoder from SigLIP.
• Extends capabilities to 201 languages of the Flores - 200.
• Achieves state - of - the - art on the Crossmodal - 3600 dataset, performing well on low - resource languages.
• This version outperforms the standard version, with results available here and here.
• There is an even better version available.

🚀 Quick Start

This model is integrated into OpenCLIP. You can use it just like any other model.

📦 Installation

!pip install -U open_clip_torch

💻 Usage Examples

Basic Usage

from open_clip import create_model_from_pretrained, get_tokenizer
from PIL import Image
import requests
import torch

model, transform = create_model_from_pretrained("nllb-clip-base-siglip", "v1", device="cuda")

tokenizer = get_tokenizer("nllb-clip-base-siglip")

class_options = ["бабочка", "butterfly", "kat"]
class_langs = ["rus_Cyrl", "eng_Latn", "afr_Latn"]

text_inputs = []
for i in range(len(class_options)):
    tokenizer.set_language(class_langs[i])
    text_inputs.append(tokenizer(class_options[i]))
text_inputs = torch.stack(text_inputs).squeeze(1).to("cuda")

image_path = "https://huggingface.co/spaces/jjourney1125/swin2sr/resolve/main/samples/butterfly.jpg"
image = Image.open(requests.get(image_path, stream=True).raw)

image_inputs = transform(image).unsqueeze(0).to("cuda")

with torch.inference_mode():
    logits_per_image, logits_per_text = model.get_logits(image_inputs, text_inputs)

print(logits_per_image.softmax(dim=-1))

📚 Documentation

You can find more details about the model in the paper.

📄 License

This model is released under the cc - by - nc - 4.0 license.

🔗 Try it in Colab

👏 Acknowledgements

I thank ML Collective for providing Google Cloud compute resources to train the OpenCLIP - compatible version of NLLB - CLIP.