Modernbert Base Gooaq

🚀 基于answerdotai/ModernBERT-base的句子转换器

这是一个基于 sentence-transformers 的模型，它在 gooaq 数据集上对 answerdotai/ModernBERT-base 进行了微调。该模型可以将句子和段落映射到一个 768 维的密集向量空间，可用于语义文本相似度计算、语义搜索、释义挖掘、文本分类、聚类等任务。

此模型使用 train_st_gooaq.py 脚本，借助 RTX 3090 进行微调，仅使用了 10GB 的显存。

🚀 快速开始

本模型可用于将句子和段落映射到 768 维的密集向量空间，在语义文本相似度、语义搜索、释义挖掘、文本分类、聚类等多个领域发挥作用。下面将介绍如何使用该模型。

安装 Sentence Transformers 库

pip install -U sentence-transformers

加载模型并进行推理

from sentence_transformers import SentenceTransformer

# 从 🤗 Hub 下载模型
model = SentenceTransformer("tomaarsen/ModernBERT-base-gooaq")
# 进行推理
sentences = [
    'are you human korean novela?',
    "Are You Human? (Korean: 너도 인간이니; RR: Neodo Inganini; lit. Are You Human Too?) is a 2018 South Korean television series starring Seo Kang-jun and Gong Seung-yeon. It aired on KBS2's Mondays and Tuesdays at 22:00 (KST) time slot, from June 4 to August 7, 2018.",
    'A relative of European pear varieties like Bartlett and Anjou, the Asian pear is great used in recipes or simply eaten out of hand. It retains a crispness that works well in slaws and salads, and it holds its shape better than European pears when baked and cooked.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# 获取嵌入向量的相似度得分
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

✨ 主要特性

• 多任务适用性：可用于语义文本相似度计算、语义搜索、释义挖掘、文本分类、聚类等多种自然语言处理任务。
• 微调优化：在 gooaq 数据集上对 answerdotai/ModernBERT-base 进行微调，提升了模型性能。
• 显存高效利用：使用 RTX 3090 进行微调时仅使用 10GB 显存。

📚 详细文档

模型详情

模型描述

模型来源

• 文档：Sentence Transformers 文档
• 仓库：GitHub 上的 Sentence Transformers
• Hugging Face：Hugging Face 上的 Sentence Transformers

完整模型架构

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: ModernBertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

评估指标

信息检索

在 NanoNQ 和 NanoMSMARCO 数据集上使用 InformationRetrievalEvaluator 进行评估。

Nano BEIR

在 NanoBEIR_mean 数据集上使用 NanoBEIREvaluator 进行评估。

训练详情

训练数据集

• 数据集：gooaq（版本 b089f72）
• 大小：3,012,496 个训练样本
• 列：question 和 answer
• 基于前 1000 个样本的近似统计信息： | | 问题 | 答案 | |------|------|------| | 类型 | 字符串 | 字符串 | | 详情 |
  • 最小值：8 个标记
  • 平均值：12.0 个标记
  • 最大值：21 个标记
  |
  • 最小值：15 个标记
  • 平均值：58.17 个标记
  • 最大值：190 个标记
  |
• 样本： | 问题 | 答案 | |------|------| | what is the difference between clay and mud mask? | The main difference between the two is that mud is a skin-healing agent, while clay is a cosmetic, drying agent. Clay masks are most useful for someone who has oily skin and is prone to breakouts of acne and blemishes. | | myki how much on card? | A full fare myki card costs $6 and a concession, seniors or child myki costs $3. For more information about how to use your myki, visit ptv.vic.gov.au or call 1800 800 007. | | how to find out if someone blocked your phone number on iphone? | If you get a notification like "Message Not Delivered" or you get no notification at all, that's a sign of a potential block. Next, you could try calling the person. If the call goes right to voicemail or rings once (or a half ring) then goes to voicemail, that's further evidence you may have been blocked. |
• 损失函数：CachedMultipleNegativesRankingLoss，参数如下：

{
    "scale": 20.0,
    "similarity_fct": "cos_sim"
}

评估数据集

• 数据集：gooaq（版本 b089f72）
• 大小：3,012,496 个评估样本
• 列：question 和 answer
• 基于前 1000 个样本的近似统计信息： | | 问题 | 答案 | |------|------|------| | 类型 | 字符串 | 字符串 | | 详情 |
  • 最小值：8 个标记
  • 平均值：12.05 个标记
  • 最大值：21 个标记
  |
  • 最小值：13 个标记
  • 平均值：59.08 个标记
  • 最大值：116 个标记
  |
• 样本： | 问题 | 答案 | |------|------| | how do i program my directv remote with my tv? | ['Press MENU on your remote.', 'Select Settings & Help > Settings > Remote Control > Program Remote.', 'Choose the device (TV, audio, DVD) you wish to program. ... ', 'Follow the on-screen prompts to complete programming.'] | | are rodrigues fruit bats nocturnal? | Before its numbers were threatened by habitat destruction, storms, and hunting, some of those groups could number 500 or more members. Sunrise, sunset. Rodrigues fruit bats are most active at dawn, at dusk, and at night. | | why does your heart rate increase during exercise bbc bitesize? | During exercise there is an increase in physical activity and muscle cells respire more than they do when the body is at rest. The heart rate increases during exercise. The rate and depth of breathing increases - this makes sure that more oxygen is absorbed into the blood, and more carbon dioxide is removed from it. |
• 损失函数：CachedMultipleNegativesRankingLoss，参数如下：

{
    "scale": 20.0,
    "similarity_fct": "cos_sim"
}

训练超参数

非默认超参数

• eval_strategy：steps
• per_device_train_batch_size：2048
• per_device_eval_batch_size：2048
• learning_rate：8e-05
• num_train_epochs：1
• warmup_ratio：0.05
• bf16：True
• batch_sampler：no_duplicates

所有超参数

训练日志

框架版本

• Python：3.11.10
• Sentence Transformers：3.3.1
• Transformers：4.48.0.dev0
• PyTorch：2.6.0.dev20241112+cu121
• Accelerate：1.2.0
• Datasets：3.2.0
• Tokenizers：0.21.0

📄 许可证

文档中未提及许可证信息。

🔧 技术细节

本模型在 answerdotai/ModernBERT-base 的基础上，使用 gooaq 数据集进行微调。微调过程中使用了 CachedMultipleNegativesRankingLoss 损失函数，并设置了特定的参数。训练过程中使用了 RTX 3090 显卡，仅使用 10GB 显存。通过对模型的微调，使其能够更好地处理句子和段落的嵌入表示，适用于多种自然语言处理任务。

📝 引用

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

CachedMultipleNegativesRankingLoss

@misc{gao2021scaling,
    title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
    author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
    year={2021},
    eprint={2101.06983},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}