Turemb 512

🚀 turemb_512

這是一個 sentence-transformers 模型，它可以將句子和段落映射到一個 512 維的密集向量空間，可用於聚類或語義搜索等任務。

🚀 快速開始

本模型有兩種使用方式，分別是使用 sentence-transformers 庫和使用 HuggingFace Transformers 庫，下面將分別介紹。

📦 安裝指南

若要使用 sentence-transformers 庫，你需要先安裝它：

pip install -U sentence-transformers

💻 使用示例

基礎用法（Sentence-Transformers）

安裝好 sentence-transformers 後，你可以按照以下方式使用模型：

from sentence_transformers import SentenceTransformer
sentences = ["This is an example sentence", "Each sentence is converted"]

model = SentenceTransformer('{MODEL_NAME}')
embeddings = model.encode(sentences)
print(embeddings)

高級用法（HuggingFace Transformers）

若不使用 sentence-transformers，你可以按如下方式使用模型：首先，將輸入數據傳入 Transformer 模型，然後對上下文詞嵌入應用正確的池化操作。

from transformers import AutoTokenizer, AutoModel
import torch


#Mean Pooling - Take attention mask into account for correct averaging
def mean_pooling(model_output, attention_mask):
    token_embeddings = model_output[0] #First element of model_output contains all token embeddings
    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)


# Sentences we want sentence embeddings for
sentences = ['This is an example sentence', 'Each sentence is converted']

# Load model from HuggingFace Hub
tokenizer = AutoTokenizer.from_pretrained('{MODEL_NAME}')
model = AutoModel.from_pretrained('{MODEL_NAME}')

# Tokenize sentences
encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')

# Compute token embeddings
with torch.no_grad():
    model_output = model(**encoded_input)

# Perform pooling. In this case, mean pooling.
sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])

print("Sentence embeddings:")
print(sentence_embeddings)

📚 詳細文檔

評估結果

若要對該模型進行自動評估，請參考 Sentence Embeddings Benchmark：https://seb.sbert.net

訓練參數

該模型的訓練參數如下：

數據加載器

使用 torch.utils.data.dataloader.DataLoader，長度為 14435，參數如下：

{'batch_size': 16, 'sampler': 'torch.utils.data.sampler.RandomSampler', 'batch_sampler': 'torch.utils.data.sampler.BatchSampler'}

損失函數

使用 sentence_transformers.losses.MultipleNegativesRankingLoss.MultipleNegativesRankingLoss，參數如下：

{'scale': 20.0, 'similarity_fct': 'cos_sim'}

fit() 方法的參數

{
    "epochs": 12,
    "evaluation_steps": 1000,
    "evaluator": "sentence_transformers.evaluation.EmbeddingSimilarityEvaluator.EmbeddingSimilarityEvaluator",
    "max_grad_norm": 1,
    "optimizer_class": "<class 'torch.optim.adamw.AdamW'>",
    "optimizer_params": {
        "lr": 0.0001
    },
    "scheduler": "WarmupLinear",
    "steps_per_epoch": null,
    "warmup_steps": 866,
    "weight_decay": 0.005
}

完整模型架構

SentenceTransformer(
  (0): Transformer({'max_seq_length': None, 'do_lower_case': False}) with Transformer model: T5EncoderModel 
  (1): Pooling({'word_embedding_dimension': 512, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False})
)

引用與作者

@article{,
  title={Translation Aligned Sentence Embeddings for Turkish Language},
  author={Unlu, Eren and Ciftci, Unver},
  journal={arXiv preprint arXiv:2311.09748},
  year={2023}
}

@article{chung2022scaling,
  title={Scaling instruction-finetuned language models},
  author={Chung, Hyung Won and Hou, Le and Longpre, Shayne and Zoph, Barret and Tay, Yi and Fedus, William and Li, Yunxuan and Wang, Xuezhi and Dehghani, Mostafa and Brahma, Siddhartha and others},
  journal={arXiv preprint arXiv:2210.11416},
  year={2022}
}

@article{budur2020data,
  title={Data and representation for turkish natural language inference},
  author={Budur, Emrah and {\"O}z{\c{c}}elik, R{\i}za and G{\"u}ng{\"o}r, Tunga and Potts, Christopher},
  journal={arXiv preprint arXiv:2004.14963},
  year={2020}
}

@article{tiedemann2020tatoeba,
  title={The Tatoeba Translation Challenge--Realistic Data Sets for Low Resource and Multilingual MT},
  author={Tiedemann, J{\"o}rg},
  journal={arXiv preprint arXiv:2010.06354},
  year={2020}
}

@article{unal2016tasviret,
  title={Tasviret: G{\"o}r{\"u}nt{\"u}lerden otomatik t{\"u}rk{\c{c}}e a{\c{c}}{\i}klama olusturma I{\c{c}}in bir denekta{\c{c}}{\i} veri k{\"u}mesi (TasvirEt: A benchmark dataset for automatic Turkish description generation from images)},
  author={Unal, Mesut Erhan and Citamak, Begum and Yagcioglu, Semih and Erdem, Aykut and Erdem, Erkut and Cinbis, Nazli Ikizler and Cakici, Ruket},
  journal={IEEE Sinyal Isleme ve Iletisim Uygulamalar{\i} Kurultay{\i} (SIU 2016)},
  year={2016}
}