Nllb Siglip Mrl Base

🚀 NLLB-SigLIP-MRL Model

The NLLB-SigLIP-MRL model combines the text encoder from the NLLB model and the image encoder from the SigLIP model. It extends the model's capabilities to 201 languages of the Flores-200.

✨ Features

• Combines NLLB text encoder and SigLIP image encoder.
• Supports 201 languages of the Flores-200.
• Trained using a variation of Matryoshka Representation learning to generate embeddings of sizes [32, 64, 128, 256, 512] in addition to the original 768.
• The full embedding model sets a new state-of-the-art for multilingual image and text retrieval on both XTD10 and Crossmodal-3600.

📦 Installation

Variable resolutions

!pip install -U transformers open_clip_torch

OpenCLIP

!pip install -U open_clip_torch

💻 Usage Examples

Basic Usage (Variable resolutions)

from transformers import AutoModel
from PIL import Image
import requests
import torch

model = AutoModel.from_pretrained("visheratin/nllb-siglip-mrl-base", device="cpu", trust_remote_code=True)

image_path = "https://huggingface.co/spaces/jjourney1125/swin2sr/resolve/main/samples/butterfly.jpg"
image = Image.open(requests.get(image_path, stream=True).raw)

class_options = ["бабочка", "butterfly", "kat"]
class_langs = ["rus_Cyrl", "eng_Latn", "afr_Latn"]

image_logits, text_logits = model.get_logits(
    images=[image],
    texts=class_options,
    langs=class_langs,
    resolution=512 # set resolution here or set `None` to use the original resolution
)

print(torch.softmax(image_logits, dim=1))

Advanced Usage (OpenCLIP)

from open_clip import create_model_from_pretrained, get_tokenizer
from PIL import Image
import requests
import torch

model, transform = create_model_from_pretrained("nllb-clip-base-siglip", "mrl", device="cuda")

tokenizer = get_tokenizer("nllb-clip-base-siglip")

class_options = ["бабочка", "butterfly", "kat"]
class_langs = ["rus_Cyrl", "eng_Latn", "afr_Latn"]

text_inputs = []
for i in range(len(class_options)):
    tokenizer.set_language(class_langs[i])
    text_inputs.append(tokenizer(class_options[i]))
text_inputs = torch.stack(text_inputs).squeeze(1).to("cuda")

image_path = "https://huggingface.co/spaces/jjourney1125/swin2sr/resolve/main/samples/butterfly.jpg"
image = Image.open(requests.get(image_path, stream=True).raw)

image_inputs = transform(image).unsqueeze(0).to("cuda")

with torch.inference_mode():
    logits_per_image, logits_per_text = model.get_logits(image_inputs, text_inputs)

print(logits_per_image.softmax(dim=-1))

📚 Documentation

The NLLB-SigLIP-MRL model combines a text encoder from the NLLB model and an image encoder from the SigLIP model. This combination allows the model to support 201 languages of the Flores-200. The model was trained using a variation of Matryoshka Representation learning to enable the generation of embeddings of different sizes.

The performance of the model on different datasets is shown in the following table:

📄 License

This model is released under the cc-by-nc-4.0 license.

Acknowledgements

I thank ML Collective for providing Google Cloud compute resources.