Rhetoribert

🚀 sentence-transformers/static-retrieval-mrl-en-v1

這是一個基於 sentence-transformers 的模型，它在 sci_gen_colbert_triplets 數據集上對 nomic-ai/nomic-embed-text-v1.5 進行了微調。該模型能夠將學術文本中的句子映射到一個 768 維的密集向量空間，映射依據是句子的修辭功能（如總結結果、表達侷限性等），可用於功能文本相似度分析、侷限性分析、修辭功能分類、聚類等任務。

🚀 快速開始

直接使用（Sentence Transformers）

首先安裝 Sentence Transformers 庫：

pip install -U sentence-transformers

然後，你可以加載此模型並進行推理：

from sentence_transformers import SentenceTransformer

# 從 🤗 Hub 下載模型
model = SentenceTransformer("KaiserML/RhetoriBERT")
# 進行推理
sentences = [
    'Surveys and interviews: Introducing excerpts from interview data',
    "Through surveys and interviews, multiliterate teachers expressed a shared belief in the importance of fostering students' ability to navigate multiple discourse communities.",
    'The authors employ a constructivist approach to learning, where students build knowledge through active engagement with multimedia texts and collaborative discussions.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# 獲取嵌入向量的相似度分數
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

✨ 主要特性

• 能夠將學術文本中的句子映射到 768 維的密集向量空間，依據是句子的修辭功能。
• 可用於功能文本相似度分析、侷限性分析、修辭功能分類、聚類等多種任務。

📦 安裝指南

安裝 Sentence Transformers 庫：

pip install -U sentence-transformers

💻 使用示例

基礎用法

from sentence_transformers import SentenceTransformer

# 從 🤗 Hub 下載模型
model = SentenceTransformer("KaiserML/RhetoriBERT")
# 進行推理
sentences = [
    'Surveys and interviews: Introducing excerpts from interview data',
    "Through surveys and interviews, multiliterate teachers expressed a shared belief in the importance of fostering students' ability to navigate multiple discourse communities.",
    'The authors employ a constructivist approach to learning, where students build knowledge through active engagement with multimedia texts and collaborative discussions.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# 獲取嵌入向量的相似度分數
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

📚 詳細文檔

模型詳情

模型描述

模型來源

• 文檔：Sentence Transformers 文檔
• 倉庫：GitHub 上的 Sentence Transformers
• Hugging Face：Hugging Face 上的 Sentence Transformers

完整模型架構

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: NomicBertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

評估

指標

信息檢索

• 數據集：SciGen-Eval-Set
• 使用 InformationRetrievalEvaluator 進行評估

訓練詳情

訓練數據集

sci_gen_colbert_triplets

• 數據集：sci_gen_colbert_triplets，版本為 44071bd
• 大小：35,934 個訓練樣本
• 列：query、positive 和 negative
• 基於前 1000 個樣本的近似統計信息： | | query | positive | negative | |------|------|------|------| | 類型 | string | string | string | | 詳情 |
  • 最小：5 個詞元
  • 平均：10.24 個詞元
  • 最大：23 個詞元
  |
  • 最小：2 個詞元
  • 平均：39.86 個詞元
  • 最大：80 個詞元
  |
  • 最小：18 個詞元
  • 平均：40.41 個詞元
  • 最大：88 個詞元
  |
• 樣本： | query | positive | negative | |------|------|------| | Previous research: highlighting negative outcomes | Despite the widespread use of seniority-based wage systems in labor contracts, previous research has highlighted their negative outcomes, such as inefficiencies and demotivating effects on workers. | This paper, published in 1974, was among the first to establish the importance of rank-order tournaments as optimal labor contracts in microeconomics. | | Synthesising sources: contrasting evidence or ideas | Despite the observed chronic enterocolitis in Interleukin-10-deficient mice, some studies suggest that this cytokine plays a protective role in intestinal inflammation in humans (Kurimoto et al., 2001). | Chronic enterocolitis developed in Interleukin-10-deficient mice, characterized by inflammatory cell infiltration, epithelial damage, and increased production of pro-inflammatory cytokines. | | Previous research: Approaches taken | Previous research on measuring patient-relevant outcomes in osteoarthritis has primarily relied on self-reported measures, such as the Western Ontario and McMaster Universities Arthritis Index (WOMAC) (Bellamy et al., 1988). | The WOMAC (Western Ontario and McMaster Universities Osteoarthritis Index) questionnaire has been widely used in physical therapy research to assess the impact of antirheumatic drug therapy on patient-reported outcomes in individuals with hip or knee osteoarthritis. |
• 損失函數：MatryoshkaLoss，參數如下：

{
    "loss": "MultipleNegativesRankingLoss",
    "matryoshka_dims": [
        768,
        384,
        256,
        128,
        64
    ],
    "matryoshka_weights": [
        1,
        1,
        1,
        1,
        1
    ],
    "n_dims_per_step": -1
}

評估數據集

sci_gen_colbert_triplets

• 數據集：sci_gen_colbert_triplets，版本為 44071bd
• 大小：4,492 個評估樣本
• 列：query、positive 和 negative
• 基於前 1000 個樣本的近似統計信息： | | query | positive | negative | |------|------|------|------| | 類型 | string | string | string | | 詳情 |
  • 最小：5 個詞元
  • 平均：10.23 個詞元
  • 最大：23 個詞元
  |
  • 最小：18 個詞元
  • 平均：39.83 個詞元
  • 最大：84 個詞元
  |
  • 最小：8 個詞元
  • 平均：39.89 個詞元
  • 最大：84 個詞元
  |
• 樣本： | query | positive | negative | |------|------|------| | Providing background information: reference to the purpose of the study | This study aimed to investigate the impact of socioeconomic status on child development, specifically focusing on cognitive, language, and social-emotional domains. | Children from high socioeconomic status families showed significantly higher IQ scores (M = 112.5, SD = 5.6) compared to children from low socioeconomic status families (M = 104.3, SD = 6.2) in the verbal IQ subtest. | | Providing background information: reference to the literature | According to previous studies using WinGX suite for small-molecule single-crystal crystallography, the optimization of crystal structures leads to improved accuracy in determining atomic coordinates. | This paper describes the WinGX suite, a powerful tool for small-molecule single-crystal crystallography that significantly advances the field of crystallography by streamlining data collection and analysis. | | General comments on the relevant literature | Polymer brushes have gained significant attention in the field of polymer science due to their unique properties, such as controlled thickness, high surface density, and tunable interfacial properties. | Despite previous reports suggesting that polymer brushes with short grafting densities exhibit poorer performance in terms of adhesion and stability compared to those with higher grafting densities (Liu et al., 2010), our results indicate that the opposite is true for certain types of polymer brushes. |
• 損失函數：MatryoshkaLoss，參數如下：

{
    "loss": "MultipleNegativesRankingLoss",
    "matryoshka_dims": [
        768,
        384,
        256,
        128,
        64
    ],
    "matryoshka_weights": [
        1,
        1,
        1,
        1,
        1
    ],
    "n_dims_per_step": -1
}

訓練超參數

非默認超參數

• eval_strategy: steps
• per_device_train_batch_size: 256
• per_device_eval_batch_size: 256
• learning_rate: 2e-05
• num_train_epochs: 10
• warmup_ratio: 0.1
• fp16: True
• load_best_model_at_end: True

所有超參數

訓練日誌

框架版本

• Python: 3.11.11
• Sentence Transformers: 3.3.1
• Transformers: 4.47.1
• PyTorch: 2.5.1+cu121
• Accelerate: 1.2.1
• Datasets: 3.2.0
• Tokenizers: 0.21.0

📄 許可證

本項目採用 apache-2.0 許可證。

📖 引用

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}