Modernbert Embed Base Legal MRL

🚀 ModernBERT Embed base Legal Matryoshka

這是一個基於 sentence-transformers 的模型，它在 AdamLucek/legal-rag-positives-synthetic 數據集上對 nomic-ai/modernbert-embed-base 進行了微調。該模型可以將句子和段落映射到一個 768 維的密集向量空間，可用於語義文本相似度計算、語義搜索、釋義挖掘、文本分類、聚類等任務。

🚀 快速開始

直接使用（Sentence Transformers）

首先安裝 Sentence Transformers 庫：

pip install -U sentence-transformers

然後，你可以加載這個模型並進行推理：

from sentence_transformers import SentenceTransformer

# 從 🤗 Hub 下載
model = SentenceTransformer("AdamLucek/ModernBERT-embed-base-legal-MRL")
# 進行推理
sentences = [
    'contracting/contracting-assistance-programs/sba-mentor-protege-program (last visited Apr. 19, \n2023). \n5 \n \nprotégé must demonstrate that the added mentor-protégé relationship will not adversely affect the \ndevelopment of either protégé firm (e.g., the second firm may not be a competitor of the first \nfirm).”  13 C.F.R. § 125.9(b)(3).',
    'What must the protégé demonstrate about the mentor-protégé relationship?',
    'What discretion do district courts have regarding a defendant’s invocation of FOIA exemptions?',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# 獲取嵌入向量的相似度得分
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

✨ 主要特性

該模型具有以下特性：

• 能夠將句子和段落映射到 768 維的密集向量空間。
• 可用於語義文本相似度、語義搜索、釋義挖掘、文本分類、聚類等多種任務。

📦 安裝指南

安裝 Sentence Transformers 庫：

pip install -U sentence-transformers

📚 詳細文檔

模型詳情

模型描述

完整模型架構

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: ModernBertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

評估

指標

信息檢索

訓練詳情

AdamLucek/legal-rag-positives-synthetic

• 數據集：AdamLucek/legal-rag-positives-synthetic
• 大小：5822 個訓練樣本
• 列：positive 和 anchor
• 基於前 1000 個樣本的近似統計信息： | | positive | anchor | | ---- | ---- | ---- | | 類型 | 字符串 | 字符串 | | 詳情 |
  • 最小：15 個標記
  • 平均：97.6 個標記
  • 最大：153 個標記
  |
  • 最小：8 個標記
  • 平均：16.68 個標記
  • 最大：41 個標記
  |
• 樣本： | positive | anchor | | ---- | ---- | | infrastructure security information,” the information at issue must, “if disclosed . . . reveal vulner-
abilities in Department of Defense critical infrastructure.” 10 U.S.C. § 130e(f). The closest the
Department comes is asserting that the information “individually or in the aggregate, would enable | What type of information must reveal vulnerabilities if disclosed? | | they have bid.” Oral Arg. Tr. at 42:18–20. Plaintiffs also assert that, should this Court require the
Polaris Solicitations to consider price at the IDIQ level, such an adjustment “adds a solicitation
requirement that would necessarily change the overall structure of the evaluation” GSA must
perform in awarding the IDIQ contracts. Oral Arg. Tr. at 43:3–5; see supra Discussion Section | Where in the document can further discussion about the assertion be found? | | otra parte. Fernández v. San Juan Cement Co., Inc., 118 DPR 713,
718-719 (1987). Nuestro más Alto Foro ha dispuesto que, la
facultad de imponer honorarios de abogados es la mejor arma que

22 Id.
23 Andamios de PR v. Newport Bonding, 179 DPR 503, 520 (2010); Pérez Rodríguez
v. López Rodríguez, supra; SLG González -Figueroa v. Pacheco Romero, supra; | What case is cited with the reference number 118 DPR 713? |
• 損失函數：MatryoshkaLoss，參數如下：

{
    "loss": "MultipleNegativesRankingLoss",
    "matryoshka_dims": [
        768,
        512,
        256,
        128,
        64
    ],
    "matryoshka_weights": [
        1,
        1,
        1,
        1,
        1
    ],
    "n_dims_per_step": -1
}

訓練超參數

非默認超參數

• eval_strategy: epoch
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 16
• gradient_accumulation_steps: 16
• learning_rate: 2e-05
• num_train_epochs: 4
• lr_scheduler_type: cosine
• warmup_ratio: 0.1
• bf16: True
• tf32: True
• load_best_model_at_end: True
• optim: adamw_torch_fused
• batch_sampler: no_duplicates

所有超參數

訓練日誌

• 加粗行表示保存的檢查點。

框架版本

• Python: 3.11.11
• Sentence Transformers: 3.3.1
• Transformers: 4.48.0
• PyTorch: 2.5.1+cu121
• Accelerate: 1.2.1
• Datasets: 3.2.0
• Tokenizers: 0.21.0

📄 許可證

本模型使用 apache-2.0 許可證。

📖 引用

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}