Turemb 512

🚀 turemb_512

这是一个 sentence-transformers 模型，它可以将句子和段落映射到一个 512 维的密集向量空间，可用于聚类或语义搜索等任务。

🚀 快速开始

本模型有两种使用方式，分别是使用 sentence-transformers 库和使用 HuggingFace Transformers 库，下面将分别介绍。

📦 安装指南

若要使用 sentence-transformers 库，你需要先安装它：

pip install -U sentence-transformers

💻 使用示例

基础用法（Sentence-Transformers）

安装好 sentence-transformers 后，你可以按照以下方式使用模型：

from sentence_transformers import SentenceTransformer
sentences = ["This is an example sentence", "Each sentence is converted"]

model = SentenceTransformer('{MODEL_NAME}')
embeddings = model.encode(sentences)
print(embeddings)

高级用法（HuggingFace Transformers）

若不使用 sentence-transformers，你可以按如下方式使用模型：首先，将输入数据传入 Transformer 模型，然后对上下文词嵌入应用正确的池化操作。

from transformers import AutoTokenizer, AutoModel
import torch


#Mean Pooling - Take attention mask into account for correct averaging
def mean_pooling(model_output, attention_mask):
    token_embeddings = model_output[0] #First element of model_output contains all token embeddings
    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)


# Sentences we want sentence embeddings for
sentences = ['This is an example sentence', 'Each sentence is converted']

# Load model from HuggingFace Hub
tokenizer = AutoTokenizer.from_pretrained('{MODEL_NAME}')
model = AutoModel.from_pretrained('{MODEL_NAME}')

# Tokenize sentences
encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')

# Compute token embeddings
with torch.no_grad():
    model_output = model(**encoded_input)

# Perform pooling. In this case, mean pooling.
sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])

print("Sentence embeddings:")
print(sentence_embeddings)

📚 详细文档

评估结果

若要对该模型进行自动评估，请参考 Sentence Embeddings Benchmark：https://seb.sbert.net

训练参数

该模型的训练参数如下：

数据加载器

使用 torch.utils.data.dataloader.DataLoader，长度为 14435，参数如下：

{'batch_size': 16, 'sampler': 'torch.utils.data.sampler.RandomSampler', 'batch_sampler': 'torch.utils.data.sampler.BatchSampler'}

损失函数

使用 sentence_transformers.losses.MultipleNegativesRankingLoss.MultipleNegativesRankingLoss，参数如下：

{'scale': 20.0, 'similarity_fct': 'cos_sim'}

fit() 方法的参数

{
    "epochs": 12,
    "evaluation_steps": 1000,
    "evaluator": "sentence_transformers.evaluation.EmbeddingSimilarityEvaluator.EmbeddingSimilarityEvaluator",
    "max_grad_norm": 1,
    "optimizer_class": "<class 'torch.optim.adamw.AdamW'>",
    "optimizer_params": {
        "lr": 0.0001
    },
    "scheduler": "WarmupLinear",
    "steps_per_epoch": null,
    "warmup_steps": 866,
    "weight_decay": 0.005
}

完整模型架构

SentenceTransformer(
  (0): Transformer({'max_seq_length': None, 'do_lower_case': False}) with Transformer model: T5EncoderModel 
  (1): Pooling({'word_embedding_dimension': 512, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False})
)

引用与作者

@article{,
  title={Translation Aligned Sentence Embeddings for Turkish Language},
  author={Unlu, Eren and Ciftci, Unver},
  journal={arXiv preprint arXiv:2311.09748},
  year={2023}
}

@article{chung2022scaling,
  title={Scaling instruction-finetuned language models},
  author={Chung, Hyung Won and Hou, Le and Longpre, Shayne and Zoph, Barret and Tay, Yi and Fedus, William and Li, Yunxuan and Wang, Xuezhi and Dehghani, Mostafa and Brahma, Siddhartha and others},
  journal={arXiv preprint arXiv:2210.11416},
  year={2022}
}

@article{budur2020data,
  title={Data and representation for turkish natural language inference},
  author={Budur, Emrah and {\"O}z{\c{c}}elik, R{\i}za and G{\"u}ng{\"o}r, Tunga and Potts, Christopher},
  journal={arXiv preprint arXiv:2004.14963},
  year={2020}
}

@article{tiedemann2020tatoeba,
  title={The Tatoeba Translation Challenge--Realistic Data Sets for Low Resource and Multilingual MT},
  author={Tiedemann, J{\"o}rg},
  journal={arXiv preprint arXiv:2010.06354},
  year={2020}
}

@article{unal2016tasviret,
  title={Tasviret: G{\"o}r{\"u}nt{\"u}lerden otomatik t{\"u}rk{\c{c}}e a{\c{c}}{\i}klama olusturma I{\c{c}}in bir denekta{\c{c}}{\i} veri k{\"u}mesi (TasvirEt: A benchmark dataset for automatic Turkish description generation from images)},
  author={Unal, Mesut Erhan and Citamak, Begum and Yagcioglu, Semih and Erdem, Aykut and Erdem, Erkut and Cinbis, Nazli Ikizler and Cakici, Ruket},
  journal={IEEE Sinyal Isleme ve Iletisim Uygulamalar{\i} Kurultay{\i} (SIU 2016)},
  year={2016}
}