Reranker ModernBERT Large Gooaq Bce

🚀 在GooAQ上训练的ModernBERT-large模型

这是一个基于answerdotai/ModernBERT-large，使用sentence-transformers库微调得到的交叉编码器模型。它可以计算文本对的得分，可用于文本重排序和语义搜索。

训练脚本请参考training_gooaq_bce.py，该脚本仅将基础模型从answerdotai/ModernBERT-base更新为answerdotai/ModernBERT-large。此脚本也在交叉编码器>训练概述文档和使用Sentence Transformers v4训练和微调重排序模型博客文章中有所描述。

🚀 快速开始

直接使用（Sentence Transformers）

首先安装Sentence Transformers库：

pip install -U sentence-transformers

然后，你可以加载此模型并进行推理：

from sentence_transformers import CrossEncoder

# 从🤗 Hub下载
model = CrossEncoder("tomaarsen/reranker-ModernBERT-large-gooaq-bce")
# 获取文本对的得分
pairs = [
    ['what are the characteristics and elements of poetry?', 'The elements of poetry include meter, rhyme, form, sound, and rhythm (timing). Different poets use these elements in many different ways.'],
    ['what are the characteristics and elements of poetry?', "What's the first rule of writing poetry? That there are no rules — it's all up to you! Of course there are different poetic forms and devices, and free verse poems are one of the many poetic styles; they have no structure when it comes to format or even rhyming."],
    ['what are the characteristics and elements of poetry?', "['Blank verse. Blank verse is poetry written with a precise meter—almost always iambic pentameter—that does not rhyme. ... ', 'Rhymed poetry. In contrast to blank verse, rhymed poems rhyme by definition, although their scheme varies. ... ', 'Free verse. ... ', 'Epics. ... ', 'Narrative poetry. ... ', 'Haiku. ... ', 'Pastoral poetry. ... ', 'Sonnet.']"],
    ['what are the characteristics and elements of poetry?', 'The main component of poetry is its meter (the regular pattern of strong and weak stress). When a poem has a recognizable but varying pattern of stressed and unstressed syllables, the poetry is written in verse. ... There are many possible patterns of verse, and the basic pattern of each unit is called a foot.'],
    ['what are the characteristics and elements of poetry?', "Some poetry may not make sense to you. But that's because poets don't write to be understood by others. They write because they must. The feelings and emotions that reside within them need to be expressed."],
]
scores = model.predict(pairs)
print(scores.shape)
# (5,)

# 或者根据与单个文本的相似度对不同文本进行排序
ranks = model.rank(
    'what are the characteristics and elements of poetry?',
    [
        'The elements of poetry include meter, rhyme, form, sound, and rhythm (timing). Different poets use these elements in many different ways.',
        "What's the first rule of writing poetry? That there are no rules — it's all up to you! Of course there are different poetic forms and devices, and free verse poems are one of the many poetic styles; they have no structure when it comes to format or even rhyming.",
        "['Blank verse. Blank verse is poetry written with a precise meter—almost always iambic pentameter—that does not rhyme. ... ', 'Rhymed poetry. In contrast to blank verse, rhymed poems rhyme by definition, although their scheme varies. ... ', 'Free verse. ... ', 'Epics. ... ', 'Narrative poetry. ... ', 'Haiku. ... ', 'Pastoral poetry. ... ', 'Sonnet.']",
        'The main component of poetry is its meter (the regular pattern of strong and weak stress). When a poem has a recognizable but varying pattern of stressed and unstressed syllables, the poetry is written in verse. ... There are many possible patterns of verse, and the basic pattern of each unit is called a foot.',
        "Some poetry may not make sense to you. But that's because poets don't write to be understood by others. They write because they must. The feelings and emotions that reside within them need to be expressed.",
    ]
)
# [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...]

✨ 主要特性

• 基于answerdotai/ModernBERT-large模型微调，具备强大的文本理解能力。
• 作为交叉编码器，能够计算文本对的得分，适用于文本重排序和语义搜索任务。
• 支持长序列输入，最大序列长度可达8192个标记。

📦 安装指南

安装Sentence Transformers库：

pip install -U sentence-transformers

💻 使用示例

基础用法

from sentence_transformers import CrossEncoder

# 从🤗 Hub下载
model = CrossEncoder("tomaarsen/reranker-ModernBERT-large-gooaq-bce")
# 获取文本对的得分
pairs = [
    ['what are the characteristics and elements of poetry?', 'The elements of poetry include meter, rhyme, form, sound, and rhythm (timing). Different poets use these elements in many different ways.'],
    ['what are the characteristics and elements of poetry?', "What's the first rule of writing poetry? That there are no rules — it's all up to you! Of course there are different poetic forms and devices, and free verse poems are one of the many poetic styles; they have no structure when it comes to format or even rhyming."],
    ['what are the characteristics and elements of poetry?', "['Blank verse. Blank verse is poetry written with a precise meter—almost always iambic pentameter—that does not rhyme. ... ', 'Rhymed poetry. In contrast to blank verse, rhymed poems rhyme by definition, although their scheme varies. ... ', 'Free verse. ... ', 'Epics. ... ', 'Narrative poetry. ... ', 'Haiku. ... ', 'Pastoral poetry. ... ', 'Sonnet.']"],
    ['what are the characteristics and elements of poetry?', 'The main component of poetry is its meter (the regular pattern of strong and weak stress). When a poem has a recognizable but varying pattern of stressed and unstressed syllables, the poetry is written in verse. ... There are many possible patterns of verse, and the basic pattern of each unit is called a foot.'],
    ['what are the characteristics and elements of poetry?', "Some poetry may not make sense to you. But that's because poets don't write to be understood by others. They write because they must. The feelings and emotions that reside within them need to be expressed."],
]
scores = model.predict(pairs)
print(scores.shape)
# (5,)

高级用法

# 根据与单个文本的相似度对不同文本进行排序
ranks = model.rank(
    'what are the characteristics and elements of poetry?',
    [
        'The elements of poetry include meter, rhyme, form, sound, and rhythm (timing). Different poets use these elements in many different ways.',
        "What's the first rule of writing poetry? That there are no rules — it's all up to you! Of course there are different poetic forms and devices, and free verse poems are one of the many poetic styles; they have no structure when it comes to format or even rhyming.",
        "['Blank verse. Blank verse is poetry written with a precise meter—almost always iambic pentameter—that does not rhyme. ... ', 'Rhymed poetry. In contrast to blank verse, rhymed poems rhyme by definition, although their scheme varies. ... ', 'Free verse. ... ', 'Epics. ... ', 'Narrative poetry. ... ', 'Haiku. ... ', 'Pastoral poetry. ... ', 'Sonnet.']",
        'The main component of poetry is its meter (the regular pattern of strong and weak stress). When a poem has a recognizable but varying pattern of stressed and unstressed syllables, the poetry is written in verse. ... There are many possible patterns of verse, and the basic pattern of each unit is called a foot.',
        "Some poetry may not make sense to you. But that's because poets don't write to be understood by others. They write because they must. The feelings and emotions that reside within them need to be expressed.",
    ]
)
# [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...]

📚 详细文档

模型详情

模型描述

模型来源

• 文档：Sentence Transformers文档
• 文档：交叉编码器文档
• 仓库：GitHub上的Sentence Transformers
• Hugging Face：Hugging Face上的交叉编码器

评估

指标

交叉编码器重排序（gooaq-dev数据集，always_rerank_positives=False）

使用CrossEncoderRerankingEvaluator进行评估，参数如下：

{
    "at_k": 10,
    "always_rerank_positives": false
}

交叉编码器重排序（gooaq-dev数据集，always_rerank_positives=True）

使用CrossEncoderRerankingEvaluator进行评估，参数如下：

{
    "at_k": 10,
    "always_rerank_positives": true
}

交叉编码器重排序（NanoMSMARCO_R100、NanoNFCorpus_R100和NanoNQ_R100数据集，always_rerank_positives=True）

使用CrossEncoderRerankingEvaluator进行评估，参数如下：

{
    "at_k": 10,
    "always_rerank_positives": true
}

交叉编码器Nano BEIR（NanoBEIR_R100_mean数据集）

使用CrossEncoderNanoBEIREvaluator进行评估，参数如下：

{
    "dataset_names": [
        "msmarco",
        "nfcorpus",
        "nq"
    ],
    "rerank_k": 100,
    "at_k": 10,
    "always_rerank_positives": true
}

训练详情

训练数据集

未命名数据集

• 大小：578,402个训练样本
• 列：question、answer和label
• 基于前1000个样本的近似统计信息： | | 问题 | 答案 | 标签 | |------|------|------|------| | 类型 | 字符串 | 字符串 | 整数 | | 详情 |
  • 最小：22个字符
  • 平均：43.99个字符
  • 最大：93个字符
  |
  • 最小：51个字符
  • 平均：252.75个字符
  • 最大：378个字符
  |
  • 0：约82.30%
  • 1：约17.70%
  |
• 样本： | 问题 | 答案 | 标签 | |------|------|------| | what are the characteristics and elements of poetry? | The elements of poetry include meter, rhyme, form, sound, and rhythm (timing). Different poets use these elements in many different ways. | 1 | | what are the characteristics and elements of poetry? | What's the first rule of writing poetry? That there are no rules — it's all up to you! Of course there are different poetic forms and devices, and free verse poems are one of the many poetic styles; they have no structure when it comes to format or even rhyming. | 0 | | what are the characteristics and elements of poetry? | ['Blank verse. Blank verse is poetry written with a precise meter—almost always iambic pentameter—that does not rhyme. ... ', 'Rhymed poetry. In contrast to blank verse, rhymed poems rhyme by definition, although their scheme varies. ... ', 'Free verse. ... ', 'Epics. ... ', 'Narrative poetry. ... ', 'Haiku. ... ', 'Pastoral poetry. ... ', 'Sonnet.'] | 0 |
• 损失函数：BinaryCrossEntropyLoss，参数如下：

{
    "activation_fn": "torch.nn.modules.linear.Identity",
    "pos_weight": 5
}

训练超参数

非默认超参数

• eval_strategy：steps
• per_device_train_batch_size：64
• per_device_eval_batch_size：64
• learning_rate：2e-05
• num_train_epochs：1
• warmup_ratio：0.1
• seed：12
• bf16：True
• dataloader_num_workers：4
• load_best_model_at_end：True

所有超参数

训练日志

加粗行表示保存的检查点。

框架版本

• Python：3.11.10
• Sentence Transformers：3.5.0.dev0
• Transformers：4.49.0
• PyTorch：2.5.1+cu124
• Accelerate：1.5.2
• Datasets：2.21.0
• Tokenizers：0.21.0

🔧 技术细节

该模型基于answerdotai/ModernBERT-large进行微调，使用BinaryCrossEntropyLoss作为损失函数，在训练过程中采用了一系列超参数进行优化，如学习率、批次大小、训练轮数等。通过在特定数据集上的训练，模型能够学习到文本对之间的语义关系，从而实现文本重排序和语义搜索的功能。

📄 许可证

本模型使用Apache-2.0许可证。

📖 引用

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}