Reranker ModernBERT Large Gooaq Bce

🚀 GooAQで訓練されたModernBERT-largeモデル

このモデルは、answerdotai/ModernBERT-largeをベースに、sentence-transformersライブラリを使って微調整されたクロスエンコーダモデルです。テキストペアのスコアを計算し、テキストの再ランキングや意味検索に利用できます。

訓練スクリプトについては、training_gooaq_bce.pyを参照してください。このスクリプトでは、ベースモデルをanswerdotai/ModernBERT-baseからanswerdotai/ModernBERT-largeに更新しています。このスクリプトは、クロスエンコーダ>訓練の概要のドキュメントや、Sentence Transformers v4を使った再ランキングモデルの訓練と微調整のブログ記事でも説明されています。

🚀 クイックスタート

Sentence Transformersを使った直接利用

まず、Sentence Transformersライブラリをインストールします。

pip install -U sentence-transformers

次に、このモデルをロードして推論を行います。

from sentence_transformers import CrossEncoder

# 🤗 Hubからダウンロード
model = CrossEncoder("tomaarsen/reranker-ModernBERT-large-gooaq-bce")
# テキストペアのスコアを取得
pairs = [
    ['what are the characteristics and elements of poetry?', 'The elements of poetry include meter, rhyme, form, sound, and rhythm (timing). Different poets use these elements in many different ways.'],
    ['what are the characteristics and elements of poetry?', "What's the first rule of writing poetry? That there are no rules — it's all up to you! Of course there are different poetic forms and devices, and free verse poems are one of the many poetic styles; they have no structure when it comes to format or even rhyming."],
    ['what are the characteristics and elements of poetry?', "['Blank verse. Blank verse is poetry written with a precise meter—almost always iambic pentameter—that does not rhyme. ... ', 'Rhymed poetry. In contrast to blank verse, rhymed poems rhyme by definition, although their scheme varies. ... ', 'Free verse. ... ', 'Epics. ... ', 'Narrative poetry. ... ', 'Haiku. ... ', 'Pastoral poetry. ... ', 'Sonnet.']"],
    ['what are the characteristics and elements of poetry?', 'The main component of poetry is its meter (the regular pattern of strong and weak stress). When a poem has a recognizable but varying pattern of stressed and unstressed syllables, the poetry is written in verse. ... There are many possible patterns of verse, and the basic pattern of each unit is called a foot.'],
    ['what are the characteristics and elements of poetry?', "Some poetry may not make sense to you. But that's because poets don't write to be understood by others. They write because they must. The feelings and emotions that reside within them need to be expressed."],
]
scores = model.predict(pairs)
print(scores.shape)
# (5,)

# または、単一のテキストとの類似度に基づいて異なるテキストをランク付けする
ranks = model.rank(
    'what are the characteristics and elements of poetry?',
    [
        'The elements of poetry include meter, rhyme, form, sound, and rhythm (timing). Different poets use these elements in many different ways.',
        "What's the first rule of writing poetry? That there are no rules — it's all up to you! Of course there are different poetic forms and devices, and free verse poems are one of the many poetic styles; they have no structure when it comes to format or even rhyming.",
        "['Blank verse. Blank verse is poetry written with a precise meter—almost always iambic pentameter—that does not rhyme. ... ', 'Rhymed poetry. In contrast to blank verse, rhymed poems rhyme by definition, although their scheme varies. ... ', 'Free verse. ... ', 'Epics. ... ', 'Narrative poetry. ... ', 'Haiku. ... ', 'Pastoral poetry. ... ', 'Sonnet.']",
        'The main component of poetry is its meter (the regular pattern of strong and weak stress). When a poem has a recognizable but varying pattern of stressed and unstressed syllables, the poetry is written in verse. ... There are many possible patterns of verse, and the basic pattern of each unit is called a foot.',
        "Some poetry may not make sense to you. But that's because poets don't write to be understood by others. They write because they must. The feelings and emotions that reside within them need to be expressed.",
    ]
)
# [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...]

✨ 主な機能

• answerdotai/ModernBERT-largeモデルをベースに微調整されており、強力なテキスト理解能力を備えています。
• クロスエンコーダとして機能し、テキストペアのスコアを計算できるため、テキストの再ランキングや意味検索タスクに適しています。
• 最大8192トークンの長いシーケンス入力をサポートしています。

📦 インストール

Sentence Transformersライブラリをインストールします。

pip install -U sentence-transformers

💻 使用例

基本的な使用法

from sentence_transformers import CrossEncoder

# 🤗 Hubからダウンロード
model = CrossEncoder("tomaarsen/reranker-ModernBERT-large-gooaq-bce")
# テキストペアのスコアを取得
pairs = [
    ['what are the characteristics and elements of poetry?', 'The elements of poetry include meter, rhyme, form, sound, and rhythm (timing). Different poets use these elements in many different ways.'],
    ['what are the characteristics and elements of poetry?', "What's the first rule of writing poetry? That there are no rules — it's all up to you! Of course there are different poetic forms and devices, and free verse poems are one of the many poetic styles; they have no structure when it comes to format or even rhyming."],
    ['what are the characteristics and elements of poetry?', "['Blank verse. Blank verse is poetry written with a precise meter—almost always iambic pentameter—that does not rhyme. ... ', 'Rhymed poetry. In contrast to blank verse, rhymed poems rhyme by definition, although their scheme varies. ... ', 'Free verse. ... ', 'Epics. ... ', 'Narrative poetry. ... ', 'Haiku. ... ', 'Pastoral poetry. ... ', 'Sonnet.']"],
    ['what are the characteristics and elements of poetry?', 'The main component of poetry is its meter (the regular pattern of strong and weak stress). When a poem has a recognizable but varying pattern of stressed and unstressed syllables, the poetry is written in verse. ... There are many possible patterns of verse, and the basic pattern of each unit is called a foot.'],
    ['what are the characteristics and elements of poetry?', "Some poetry may not make sense to you. But that's because poets don't write to be understood by others. They write because they must. The feelings and emotions that reside within them need to be expressed."],
]
scores = model.predict(pairs)
print(scores.shape)
# (5,)

高度な使用法

# 単一のテキストとの類似度に基づいて異なるテキストをランク付けする
ranks = model.rank(
    'what are the characteristics and elements of poetry?',
    [
        'The elements of poetry include meter, rhyme, form, sound, and rhythm (timing). Different poets use these elements in many different ways.',
        "What's the first rule of writing poetry? That there are no rules — it's all up to you! Of course there are different poetic forms and devices, and free verse poems are one of the many poetic styles; they have no structure when it comes to format or even rhyming.",
        "['Blank verse. Blank verse is poetry written with a precise meter—almost always iambic pentameter—that does not rhyme. ... ', 'Rhymed poetry. In contrast to blank verse, rhymed poems rhyme by definition, although their scheme varies. ... ', 'Free verse. ... ', 'Epics. ... ', 'Narrative poetry. ... ', 'Haiku. ... ', 'Pastoral poetry. ... ', 'Sonnet.']",
        'The main component of poetry is its meter (the regular pattern of strong and weak stress). When a poem has a recognizable but varying pattern of stressed and unstressed syllables, the poetry is written in verse. ... There are many possible patterns of verse, and the basic pattern of each unit is called a foot.',
        "Some poetry may not make sense to you. But that's because poets don't write to be understood by others. They write because they must. The feelings and emotions that reside within them need to be expressed.",
    ]
)
# [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...]

📚 ドキュメント

モデルの詳細

モデルの説明

モデルの出所

• ドキュメント：Sentence Transformersドキュメント
• ドキュメント：クロスエンコーダドキュメント
• リポジトリ：GitHub上のSentence Transformers
• Hugging Face：Hugging Face上のクロスエンコーダ

評価

指標

クロスエンコーダの再ランキング（gooaq-devデータセット、always_rerank_positives=False）

CrossEncoderRerankingEvaluatorを使って評価します。パラメータは以下の通りです。

{
    "at_k": 10,
    "always_rerank_positives": false
}

クロスエンコーダの再ランキング（gooaq-devデータセット、always_rerank_positives=True）

CrossEncoderRerankingEvaluatorを使って評価します。パラメータは以下の通りです。

{
    "at_k": 10,
    "always_rerank_positives": true
}

クロスエンコーダの再ランキング（NanoMSMARCO_R100、NanoNFCorpus_R100、NanoNQ_R100データセット、always_rerank_positives=True）

CrossEncoderRerankingEvaluatorを使って評価します。パラメータは以下の通りです。

{
    "at_k": 10,
    "always_rerank_positives": true
}

クロスエンコーダのNano BEIR（NanoBEIR_R100_meanデータセット）

CrossEncoderNanoBEIREvaluatorを使って評価します。パラメータは以下の通りです。

{
    "dataset_names": [
        "msmarco",
        "nfcorpus",
        "nq"
    ],
    "rerank_k": 100,
    "at_k": 10,
    "always_rerank_positives": true
}

訓練の詳細

訓練データセット

未命名データセット

• サイズ：578,402個の訓練サンプル
• 列：question、answer、label
• 最初の1000個のサンプルに基づく近似統計情報： | | 質問 | 回答 | ラベル | |------|------|------|------| | タイプ | 文字列 | 文字列 | 整数 | | 詳細 |
  • 最小：22文字
  • 平均：43.99文字
  • 最大：93文字
  |
  • 最小：51文字
  • 平均：252.75文字
  • 最大：378文字
  |
  • 0：約82.30%
  • 1：約17.70%
  |
• サンプル： | 質問 | 回答 | ラベル | |------|------|------| | what are the characteristics and elements of poetry? | The elements of poetry include meter, rhyme, form, sound, and rhythm (timing). Different poets use these elements in many different ways. | 1 | | what are the characteristics and elements of poetry? | What's the first rule of writing poetry? That there are no rules — it's all up to you! Of course there are different poetic forms and devices, and free verse poems are one of the many poetic styles; they have no structure when it comes to format or even rhyming. | 0 | | what are the characteristics and elements of poetry? | ['Blank verse. Blank verse is poetry written with a precise meter—almost always iambic pentameter—that does not rhyme. ... ', 'Rhymed poetry. In contrast to blank verse, rhymed poems rhyme by definition, although their scheme varies. ... ', 'Free verse. ... ', 'Epics. ... ', 'Narrative poetry. ... ', 'Haiku. ... ', 'Pastoral poetry. ... ', 'Sonnet.'] | 0 |
• 損失関数：BinaryCrossEntropyLoss。パラメータは以下の通りです。

{
    "activation_fn": "torch.nn.modules.linear.Identity",
    "pos_weight": 5
}

訓練のハイパーパラメータ

デフォルトではないハイパーパラメータ

• eval_strategy：steps
• per_device_train_batch_size：64
• per_device_eval_batch_size：64
• learning_rate：2e-05
• num_train_epochs：1
• warmup_ratio：0.1
• seed：12
• bf16：True
• dataloader_num_workers：4
• load_best_model_at_end：True

すべてのハイパーパラメータ

訓練ログ

太字の行は保存されたチェックポイントを表します。

フレームワークのバージョン

• Python：3.11.10
• Sentence Transformers：3.5.0.dev0
• Transformers：4.49.0
• PyTorch：2.5.1+cu124
• Accelerate：1.5.2
• Datasets：2.21.0
• Tokenizers：0.21.0

🔧 技術詳細

このモデルは、answerdotai/ModernBERT-largeをベースに微調整されており、損失関数としてBinaryCrossEntropyLossを使用しています。訓練中は、学習率、バッチサイズ、エポック数などの一連のハイパーパラメータを最適化しています。特定のデータセットでの訓練により、モデルはテキストペア間の意味関係を学習し、テキストの再ランキングや意味検索機能を実現します。

📄 ライセンス

このモデルはApache-2.0ライセンスで提供されています。

📖 引用

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}