Arabic English Sts Matryoshka V2.0

🚀 SentenceTransformer based on FacebookAI/xlm-roberta-large

This project presents a bilingual (Arabic-English) sentence-transformer model. It maps sentences and paragraphs into a 1024 - dimensional dense vector space, offering solutions for semantic textual similarity, semantic search, and other related tasks. The v2.0 version shows improved performance, especially in ar - en metrics.

🚀 Quick Start

Direct Usage (Sentence Transformers)

First, you need to install the Sentence Transformers library:

pip install -U sentence-transformers

Then, you can load the model and perform inference:

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
matryoshka_dim = 786
model = SentenceTransformer("omarelshehy/arabic-english-sts-matryoshka-v2.0", truncate_dim=matryoshka_dim)
# Run inference
sentences = [
    "She enjoyed reading books by the window as the rain poured outside.",
    "كانت تستمتع بقراءة الكتب بجانب النافذة بينما كانت الأمطار تتساقط في الخارج.",
    "Reading by the window was her favorite thing, especially during rainy days."
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

✨ Features

• Bilingual Support: Handles both Arabic and English, either separately or interchangeably.
• Matryoshka Embeddings: Supports truncation of embeddings into sizes like 1024, 768, 512, 256, 128, and 64 to optimize performance and memory usage.
• Versatile Applications: Can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, etc.

📚 Documentation

Model description

This is a Bilingual (Arabic - English) sentence - transformers model finetuned from [FacebookAI/xlm - roberta - large](https://huggingface.co/FacebookAI/xlm - roberta - large). It maps sentences & paragraphs to a 1024 - dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

The model handles both languages separately 🌐, but also interchangeably, which unlocks flexible applications for developers and researchers who want to further build on Arabic models! 💡

Matryoshka Embeddings 🪆

This model supports Matryoshka embeddings, allowing you to truncate embeddings into smaller sizes to optimize performance and memory usage, based on your task requirements. Available truncation sizes include: 1024, 768, 512, 256, 128, and 64

You can select the appropriate embedding size for your use case, ensuring flexibility in resource management.

Metrics

• Metrics (MTEB): pearson_cosine, spearman_cosine, pearson_manhattan, spearman_manhattan, pearson_euclidean, spearman_euclidean, pearson_dot, spearman_dot, pearson_max, spearman_max
• Model Index Results:
  • MTEB STS17 (en - en):
    • cosine_pearson: 87.38302667611983
    • cosine_spearman: 86.87900209442004
    • ...
  • MTEB STS17 (ar - ar):
    • cosine_pearson: 83.63516310524058
    • cosine_spearman: 83.77655124170212
    • ...
  • MTEB STS17 (en - ar):
    • cosine_pearson: 82.29919720659755
    • cosine_spearman: 82.18717939041626
    • ...

📄 License

No license information is provided in the original document.

📚 Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}