Labse Sentence Embeddings

🚀 LaBSE

A copy of setu4993/LaBSE that returns sentence embeddings (pooler_output) and implements caching. It's useful for getting multilingual sentence embeddings and bi - text retrieval.

🚀 Quick Start

This is a BERT - based model called Language - agnostic BERT Sentence Encoder (LaBSE), trained for sentence embedding across 109 languages. The pre - training combines masked language modeling with translation language modeling.

✨ Features

• Multilingual Support: Supports 109 languages including af, am, ar, etc.
• Sentence Embedding: Useful for getting multilingual sentence embeddings and bi - text retrieval.
• Equivalent Output: The embeddings produced by both the TF Hub v2 model and this PyTorch version are equivalent.

📦 Installation

No specific installation steps are provided in the original document.

💻 Usage Examples

Basic Usage

import torch
from transformers import BertModel, BertTokenizerFast


tokenizer = BertTokenizerFast.from_pretrained("setu4993/LaBSE")
model = BertModel.from_pretrained("setu4993/LaBSE")
model = model.eval()

english_sentences = [
    "dog",
    "Puppies are nice.",
    "I enjoy taking long walks along the beach with my dog.",
]
english_inputs = tokenizer(english_sentences, return_tensors="pt", padding=True)

with torch.no_grad():
    english_outputs = model(**english_inputs)

Advanced Usage

Get Sentence Embeddings

english_embeddings = english_outputs.pooler_output

Output for Other Languages

italian_sentences = [
    "cane",
    "I cuccioli sono carini.",
    "Mi piace fare lunghe passeggiate lungo la spiaggia con il mio cane.",
]
japanese_sentences = ["犬", "子犬はいいです", "私は犬と一緒にビーチを散歩するのが好きです"]
italian_inputs = tokenizer(italian_sentences, return_tensors="pt", padding=True)
japanese_inputs = tokenizer(japanese_sentences, return_tensors="pt", padding=True)

with torch.no_grad():
    italian_outputs = model(**italian_inputs)
    japanese_outputs = model(**japanese_inputs)

italian_embeddings = italian_outputs.pooler_output
japanese_embeddings = japanese_outputs.pooler_output

Calculate Sentence Similarity

import torch.nn.functional as F


def similarity(embeddings_1, embeddings_2):
    normalized_embeddings_1 = F.normalize(embeddings_1, p=2)
    normalized_embeddings_2 = F.normalize(embeddings_2, p=2)
    return torch.matmul(
        normalized_embeddings_1, normalized_embeddings_2.transpose(0, 1)
    )


print(similarity(english_embeddings, italian_embeddings))
print(similarity(english_embeddings, japanese_embeddings))
print(similarity(italian_embeddings, japanese_embeddings))

📚 Documentation

• Model: HuggingFace's model hub.
• Paper: arXiv.
• Original model: TensorFlow Hub.
• Blog post: Google AI Blog.
• Conversion from TensorFlow to PyTorch: GitHub.

This model is migrated from the v2 model on the TF Hub, which uses dict - based input. The embeddings produced by both the versions of the model are equivalent.

🔧 Technical Details

Details about data, training, evaluation and performance metrics are available in the original paper.

📄 License

This project is licensed under the apache - 2.0 license.

BibTeX entry and citation info

@misc{feng2020languageagnostic,
      title={Language-agnostic BERT Sentence Embedding},
      author={Fangxiaoyu Feng and Yinfei Yang and Daniel Cer and Naveen Arivazhagan and Wei Wang},
      year={2020},
      eprint={2007.01852},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}