Modernbert Base Gooaq

🚀 基於answerdotai/ModernBERT-base的句子轉換器

這是一個基於 sentence-transformers 的模型，它在 gooaq 數據集上對 answerdotai/ModernBERT-base 進行了微調。該模型可以將句子和段落映射到一個 768 維的密集向量空間，可用於語義文本相似度計算、語義搜索、釋義挖掘、文本分類、聚類等任務。

此模型使用 train_st_gooaq.py 腳本，藉助 RTX 3090 進行微調，僅使用了 10GB 的顯存。

🚀 快速開始

本模型可用於將句子和段落映射到 768 維的密集向量空間，在語義文本相似度、語義搜索、釋義挖掘、文本分類、聚類等多個領域發揮作用。下面將介紹如何使用該模型。

安裝 Sentence Transformers 庫

pip install -U sentence-transformers

加載模型並進行推理

from sentence_transformers import SentenceTransformer

# 從 🤗 Hub 下載模型
model = SentenceTransformer("tomaarsen/ModernBERT-base-gooaq")
# 進行推理
sentences = [
    'are you human korean novela?',
    "Are You Human? (Korean: 너도 인간이니; RR: Neodo Inganini; lit. Are You Human Too?) is a 2018 South Korean television series starring Seo Kang-jun and Gong Seung-yeon. It aired on KBS2's Mondays and Tuesdays at 22:00 (KST) time slot, from June 4 to August 7, 2018.",
    'A relative of European pear varieties like Bartlett and Anjou, the Asian pear is great used in recipes or simply eaten out of hand. It retains a crispness that works well in slaws and salads, and it holds its shape better than European pears when baked and cooked.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# 獲取嵌入向量的相似度得分
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

✨ 主要特性

• 多任務適用性：可用於語義文本相似度計算、語義搜索、釋義挖掘、文本分類、聚類等多種自然語言處理任務。
• 微調優化：在 gooaq 數據集上對 answerdotai/ModernBERT-base 進行微調，提升了模型性能。
• 顯存高效利用：使用 RTX 3090 進行微調時僅使用 10GB 顯存。

📚 詳細文檔

模型詳情

模型描述

模型來源

• 文檔：Sentence Transformers 文檔
• 倉庫：GitHub 上的 Sentence Transformers
• Hugging Face：Hugging Face 上的 Sentence Transformers

完整模型架構

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: ModernBertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

評估指標

信息檢索

在 NanoNQ 和 NanoMSMARCO 數據集上使用 InformationRetrievalEvaluator 進行評估。

Nano BEIR

在 NanoBEIR_mean 數據集上使用 NanoBEIREvaluator 進行評估。

訓練詳情

訓練數據集

• 數據集：gooaq（版本 b089f72）
• 大小：3,012,496 個訓練樣本
• 列：question 和 answer
• 基於前 1000 個樣本的近似統計信息： | | 問題 | 答案 | |------|------|------| | 類型 | 字符串 | 字符串 | | 詳情 |
  • 最小值：8 個標記
  • 平均值：12.0 個標記
  • 最大值：21 個標記
  |
  • 最小值：15 個標記
  • 平均值：58.17 個標記
  • 最大值：190 個標記
  |
• 樣本： | 問題 | 答案 | |------|------| | what is the difference between clay and mud mask? | The main difference between the two is that mud is a skin-healing agent, while clay is a cosmetic, drying agent. Clay masks are most useful for someone who has oily skin and is prone to breakouts of acne and blemishes. | | myki how much on card? | A full fare myki card costs $6 and a concession, seniors or child myki costs $3. For more information about how to use your myki, visit ptv.vic.gov.au or call 1800 800 007. | | how to find out if someone blocked your phone number on iphone? | If you get a notification like "Message Not Delivered" or you get no notification at all, that's a sign of a potential block. Next, you could try calling the person. If the call goes right to voicemail or rings once (or a half ring) then goes to voicemail, that's further evidence you may have been blocked. |
• 損失函數：CachedMultipleNegativesRankingLoss，參數如下：

{
    "scale": 20.0,
    "similarity_fct": "cos_sim"
}

評估數據集

• 數據集：gooaq（版本 b089f72）
• 大小：3,012,496 個評估樣本
• 列：question 和 answer
• 基於前 1000 個樣本的近似統計信息： | | 問題 | 答案 | |------|------|------| | 類型 | 字符串 | 字符串 | | 詳情 |
  • 最小值：8 個標記
  • 平均值：12.05 個標記
  • 最大值：21 個標記
  |
  • 最小值：13 個標記
  • 平均值：59.08 個標記
  • 最大值：116 個標記
  |
• 樣本： | 問題 | 答案 | |------|------| | how do i program my directv remote with my tv? | ['Press MENU on your remote.', 'Select Settings & Help > Settings > Remote Control > Program Remote.', 'Choose the device (TV, audio, DVD) you wish to program. ... ', 'Follow the on-screen prompts to complete programming.'] | | are rodrigues fruit bats nocturnal? | Before its numbers were threatened by habitat destruction, storms, and hunting, some of those groups could number 500 or more members. Sunrise, sunset. Rodrigues fruit bats are most active at dawn, at dusk, and at night. | | why does your heart rate increase during exercise bbc bitesize? | During exercise there is an increase in physical activity and muscle cells respire more than they do when the body is at rest. The heart rate increases during exercise. The rate and depth of breathing increases - this makes sure that more oxygen is absorbed into the blood, and more carbon dioxide is removed from it. |
• 損失函數：CachedMultipleNegativesRankingLoss，參數如下：

{
    "scale": 20.0,
    "similarity_fct": "cos_sim"
}

訓練超參數

非默認超參數

• eval_strategy：steps
• per_device_train_batch_size：2048
• per_device_eval_batch_size：2048
• learning_rate：8e-05
• num_train_epochs：1
• warmup_ratio：0.05
• bf16：True
• batch_sampler：no_duplicates

所有超參數

訓練日誌

框架版本

• Python：3.11.10
• Sentence Transformers：3.3.1
• Transformers：4.48.0.dev0
• PyTorch：2.6.0.dev20241112+cu121
• Accelerate：1.2.0
• Datasets：3.2.0
• Tokenizers：0.21.0

📄 許可證

文檔中未提及許可證信息。

🔧 技術細節

本模型在 answerdotai/ModernBERT-base 的基礎上，使用 gooaq 數據集進行微調。微調過程中使用了 CachedMultipleNegativesRankingLoss 損失函數，並設置了特定的參數。訓練過程中使用了 RTX 3090 顯卡，僅使用 10GB 顯存。通過對模型的微調，使其能夠更好地處理句子和段落的嵌入表示，適用於多種自然語言處理任務。

📝 引用

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

CachedMultipleNegativesRankingLoss

@misc{gao2021scaling,
    title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
    author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
    year={2021},
    eprint={2101.06983},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}