🚀 LEALLA-base
LEALLA是一組輕量級、與語言無關的句子嵌入模型,支持109種語言。它從LaBSE模型中提煉而來,可用於獲取多語言句子嵌入和雙語文本檢索。
🚀 快速開始
LEALLA是一個支持109種語言的輕量級語言無關句子嵌入模型集合,從LaBSE中蒸餾而來。該模型可用於獲取多語言句子嵌入和進行雙語文本檢索。
此模型從TF Hub上的v1模型遷移而來。兩個版本模型生成的嵌入是等效的。不過,對於某些語言(如日語),LEALLA模型在比較嵌入和相似度時似乎需要更高的容差。
✨ 主要特性
- 多語言支持:支持多達109種語言,包括但不限於英語、意大利語、日語等。
- 輕量級:基於知識蒸餾技術,模型輕量,減少推理速度和計算開銷。
- 語言無關:能夠生成與語言無關的句子嵌入,適用於跨語言任務。
📦 安裝指南
文檔未提及具體安裝步驟,可參考以下通用步驟:
pip install transformers torch
💻 使用示例
基礎用法
import torch
from transformers import BertModel, BertTokenizerFast
tokenizer = BertTokenizerFast.from_pretrained("setu4993/LEALLA-base")
model = BertModel.from_pretrained("setu4993/LEALLA-base")
model = model.eval()
english_sentences = [
"dog",
"Puppies are nice.",
"I enjoy taking long walks along the beach with my dog.",
]
english_inputs = tokenizer(english_sentences, return_tensors="pt", padding=True)
with torch.no_grad():
english_outputs = model(**english_inputs)
獲取句子嵌入
english_embeddings = english_outputs.pooler_output
其他語言輸出示例
italian_sentences = [
"cane",
"I cuccioli sono carini.",
"Mi piace fare lunghe passeggiate lungo la spiaggia con il mio cane.",
]
japanese_sentences = ["犬", "子犬はいいです", "私は犬と一緒にビーチを散歩するのが好きです"]
italian_inputs = tokenizer(italian_sentences, return_tensors="pt", padding=True)
japanese_inputs = tokenizer(japanese_sentences, return_tensors="pt", padding=True)
with torch.no_grad():
italian_outputs = model(**italian_inputs)
japanese_outputs = model(**japanese_inputs)
italian_embeddings = italian_outputs.pooler_output
japanese_embeddings = japanese_outputs.pooler_output
計算句子相似度
import torch.nn.functional as F
def similarity(embeddings_1, embeddings_2):
normalized_embeddings_1 = F.normalize(embeddings_1, p=2)
normalized_embeddings_2 = F.normalize(embeddings_2, p=2)
return torch.matmul(
normalized_embeddings_1, normalized_embeddings_2.transpose(0, 1)
)
print(similarity(english_embeddings, italian_embeddings))
print(similarity(english_embeddings, japanese_embeddings))
print(similarity(italian_embeddings, japanese_embeddings))
📚 詳細文檔
有關數據、訓練、評估和性能指標的詳細信息,請參閱原始論文。
🔧 技術細節
BibTeX引用和引用信息
@inproceedings{mao-nakagawa-2023-lealla,
title = "{LEALLA}: Learning Lightweight Language-agnostic Sentence Embeddings with Knowledge Distillation",
author = "Mao, Zhuoyuan and
Nakagawa, Tetsuji",
booktitle = "Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics",
month = may,
year = "2023",
address = "Dubrovnik, Croatia",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2023.eacl-main.138",
doi = "10.18653/v1/2023.eacl-main.138",
pages = "1886--1894",
abstract = "Large-scale language-agnostic sentence embedding models such as LaBSE (Feng et al., 2022) obtain state-of-the-art performance for parallel sentence alignment. However, these large-scale models can suffer from inference speed and computation overhead. This study systematically explores learning language-agnostic sentence embeddings with lightweight models. We demonstrate that a thin-deep encoder can construct robust low-dimensional sentence embeddings for 109 languages. With our proposed distillation methods, we achieve further improvements by incorporating knowledge from a teacher model. Empirical results on Tatoeba, United Nations, and BUCC show the effectiveness of our lightweight models. We release our lightweight language-agnostic sentence embedding models LEALLA on TensorFlow Hub.",
}
📄 許可證
本項目採用Apache-2.0許可證。
屬性 |
詳情 |
模型類型 |
輕量級語言無關句子嵌入模型 |
訓練數據 |
CommonCrawl、Wikipedia |