Modernbert Embed Base Legal Matryoshka 2

🚀 ModernBERT Embed base Legal Matryoshka

這是一個基於 Sentence Transformers 庫微調的模型，它在 json 數據集上對 nomic-ai/modernbert-embed-base 進行了微調。該模型能夠將句子和段落映射到 768 維的密集向量空間，可用於語義文本相似度計算、語義搜索、釋義挖掘、文本分類、聚類等任務。

🚀 快速開始

直接使用（Sentence Transformers）

首先，安裝 Sentence Transformers 庫：

pip install -U sentence-transformers

然後，你可以加載此模型並進行推理：

from sentence_transformers import SentenceTransformer

# 從 🤗 Hub 下載
model = SentenceTransformer("manishh16/modernbert-embed-base-legal-matryoshka-2")
# 進行推理
sentences = [
    'protests pursuant to 28 U.S.C. § 1491(b).  See 28 U.S.C. § 1491(b).  Section 1491(b)(1) grants the \n17 \n \ncourt jurisdiction over protests filed “by an interested party objecting to a solicitation by a Federal \nagency for bids or proposals for a proposed contract . . . or any alleged violation of statute or',
    'Under which U.S. Code section are the protests filed?',
    "Which agency's declaration is mentioned?",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# 獲取嵌入向量的相似度分數
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

✨ 主要特性

• 高維度映射：能夠將句子和段落映射到 768 維的密集向量空間，有助於更準確地捕捉語義信息。
• 長序列處理：最大序列長度可達 8192 個標記，適合處理較長的文本。
• 多任務適用：可用於語義文本相似度計算、語義搜索、釋義挖掘、文本分類、聚類等多種自然語言處理任務。

📦 安裝指南

要使用該模型，你需要安裝 Sentence Transformers 庫，可通過以下命令進行安裝：

pip install -U sentence-transformers

💻 使用示例

基礎用法

from sentence_transformers import SentenceTransformer

# 從 🤗 Hub 下載
model = SentenceTransformer("manishh16/modernbert-embed-base-legal-matryoshka-2")
# 示例句子
sentences = [
    'protests pursuant to 28 U.S.C. § 1491(b).  See 28 U.S.C. § 1491(b).  Section 1491(b)(1) grants the \n17 \n \ncourt jurisdiction over protests filed “by an interested party objecting to a solicitation by a Federal \nagency for bids or proposals for a proposed contract . . . or any alleged violation of statute or',
    'Under which U.S. Code section are the protests filed?',
    "Which agency's declaration is mentioned?",
]
# 編碼句子
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# 獲取嵌入向量的相似度分數
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

📚 詳細文檔

模型詳情

模型描述

模型來源

• 文檔：Sentence Transformers 文檔
• 倉庫：Sentence Transformers GitHub 倉庫
• Hugging Face：Sentence Transformers on Hugging Face

完整模型架構

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: ModernBertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

評估

信息檢索評估指標

以下是在不同數據集上的評估結果：

數據集：dim_768

使用 InformationRetrievalEvaluator 進行評估，參數如下：

{
    "truncate_dim": 768
}

數據集：dim_512

使用 InformationRetrievalEvaluator 進行評估，參數如下：

{
    "truncate_dim": 512
}

數據集：dim_256

使用 InformationRetrievalEvaluator 進行評估，參數如下：

{
    "truncate_dim": 256
}

數據集：dim_128

使用 InformationRetrievalEvaluator 進行評估，參數如下：

{
    "truncate_dim": 128
}

數據集：dim_64

使用 InformationRetrievalEvaluator 進行評估，參數如下：

{
    "truncate_dim": 64
}

訓練詳情

訓練數據集

json 數據集

• 數據集：json
• 大小：5822 個訓練樣本
• 列：positive 和 anchor
• 基於前 1000 個樣本的近似統計信息： | | positive | anchor | | :-------- | :------------------------------------------------------------------------------------ | :---------------------------------------------------------------------------------- | | 類型 | 字符串 | 字符串 | | 詳情 |
  • 最小值：26 個標記
  • 平均值：96.76 個標記
  • 最大值：156 個標記
  |
  • 最小值：8 個標記
  • 平均值：16.59 個標記
  • 最大值：49 個標記
  |
• 樣本示例： | positive | anchor | | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------------------------------- | | properly authenticated. See id. at 367, 19 A.3d at 429 (Harrell, J., dissenting).
Four years later, in Sublet, 442 Md. at 637-38, 113 A.3d at 697-98, we adopted the
reasonable juror test for social media evidence and applied it in the three cases that were
consolidated for purposes of the opinion: Sublet v. State, Harris v. State, and Monge- | How many years after the dissent did the adoption of the reasonable juror test occur? | | to (1) a public-interest fee waiver, (2) the expedited processing of a request, or (3) the release of
information that implicates personal privacy, all are personal to a requester and thus cannot be
assigned. See, e.g., RTC Commercial Loan Trust 1995-NP1A v. Winthrop Mgmt., 923 F. Supp.
83, 88 (E.D. Va. 1996) (holding that “certain rights are purely personal and cannot be assigned”). | What type of fee waiver is mentioned as being personal to a requester? | | ‘IRO’] staff that reviews Agency records and makes public release determinations with an eye
toward evaluating directorate-specific equities.” Id. ¶ 4. Ms. Meeks also explains that “records
frequently involve the equities of multiple directorates,” and “[w]hen records implicate the
operational interests of multiple directorates, the reviews are conducted by the relevant IROs | Who conducts the reviews when the records implicate the operational interests of multiple directorates? |

訓練損失函數

使用 MatryoshkaLoss 損失函數，參數如下：

{
    "loss": "MultipleNegativesRankingLoss",
    "matryoshka_dims": [
        768,
        512,
        256,
        128,
        64
    ],
    "matryoshka_weights": [
        1,
        1,
        1,
        1,
        1
    ],
    "n_dims_per_step": -1
}

訓練超參數

非默認超參數

• eval_strategy: epoch
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 16
• gradient_accumulation_steps: 16
• learning_rate: 2e-05
• num_train_epochs: 4
• lr_scheduler_type: cosine
• warmup_ratio: 0.1
• bf16: True
• tf32: False
• load_best_model_at_end: True
• optim: adamw_torch_fused
• batch_sampler: no_duplicates

所有超參數

訓練日誌

加粗行表示保存的檢查點。

框架版本

• Python: 3.11.11
• Sentence Transformers: 4.0.1
• Transformers: 4.50.2
• PyTorch: 2.6.0+cu124
• Accelerate: 1.5.2
• Datasets: 3.5.0
• Tokenizers: 0.21.1

📄 許可證

該模型使用 Apache-2.0 許可證。

📚 引用

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}