Reranker Bert Tiny Gooaq Bce Tanh V3

🚀 BERT-tiny在GooAQ上訓練的模型

這是一個基於Cross Encoder的模型，它使用sentence-transformers庫從prajjwal1/bert-tiny微調而來。該模型可以為文本對計算得分，可用於語義文本相似度、語義搜索、釋義挖掘、文本分類、聚類等任務。

此模型使用train_script.py進行訓練。

🚀 快速開始

本模型是一個基於Cross Encoder的微調模型，可用於計算文本對的得分，適用於多種自然語言處理任務。下面將介紹如何安裝依賴庫並使用該模型進行推理。

✨ 主要特性

• 多任務適用性：可用於語義文本相似度、語義搜索、釋義挖掘、文本分類、聚類等多種任務。
• 微調模型：基於prajjwal1/bert-tiny進行微調，能更好地適應特定任務。

📦 安裝指南

首先，你需要安裝Sentence Transformers庫：

pip install -U sentence-transformers

💻 使用示例

基礎用法

安裝好庫後，你可以加載模型並進行推理：

from sentence_transformers import CrossEncoder

# 從🤗 Hub下載模型
model = CrossEncoder("cross-encoder-testing/reranker-bert-tiny-gooaq-bce")
# 定義文本對
pairs = [
    ['are javascript developers in demand?', "JavaScript is the skill that is most in-demand for IT in 2020, according to a report from developer skills tester DevSkiller. The report, “Top IT Skills report 2020: Demand and Hiring Trends,” has JavaScript switching places with Java when compared to last year's report, with Java in third place this year, behind SQL."],
    ['are javascript developers in demand?', 'In one line difference between the two is: JavaScript is the programming language where as AngularJS is a framework based on JavaScript. ... It is also the basic for all java script based technologies like jquery, angular JS, bootstrap JS and so on. Angular JS is a framework written in javascript and uses MVC architecture.'],
    ['are javascript developers in demand?', 'Java applications are run in a virtual machine or web browser while JavaScript is run on a web browser. Java code is compiled whereas while JavaScript code is in text and in a web page. JavaScript is an OOP scripting language, whereas Java is an OOP programming language.'],
    ['are javascript developers in demand?', 'Things in the body tag are the things that should be displayed: the actual content. Javascript in the body is executed as it is read and as the page is rendered. Javascript in the head is interpreted before anything is rendered.'],
    ['are javascript developers in demand?', 'Web apps tend to be built using JavaScript, CSS and HTML5. Unlike mobile apps, there is no standard software development kit for building web apps. However, developers do have access to templates. Compared to mobile apps, web apps are usually quicker and easier to build — but they are much simpler in terms of features.'],
]
# 預測得分
scores = model.predict(pairs)
print(scores.shape)
# (5,)

# 或者根據與單個文本的相似度對不同文本進行排序
ranks = model.rank(
    'are javascript developers in demand?',
    [
        "JavaScript is the skill that is most in-demand for IT in 2020, according to a report from developer skills tester DevSkiller. The report, “Top IT Skills report 2020: Demand and Hiring Trends,” has JavaScript switching places with Java when compared to last year's report, with Java in third place this year, behind SQL.",
        'In one line difference between the two is: JavaScript is the programming language where as AngularJS is a framework based on JavaScript. ... It is also the basic for all java script based technologies like jquery, angular JS, bootstrap JS and so on. Angular JS is a framework written in javascript and uses MVC architecture.',
        'Java applications are run in a virtual machine or web browser while JavaScript is run on a web browser. Java code is compiled whereas while JavaScript code is in text and in a web page. JavaScript is an OOP scripting language, whereas Java is an OOP programming language.',
        'Things in the body tag are the things that should be displayed: the actual content. Javascript in the body is executed as it is read and as the page is rendered. Javascript in the head is interpreted before anything is rendered.',
        'Web apps tend to be built using JavaScript, CSS and HTML5. Unlike mobile apps, there is no standard software development kit for building web apps. However, developers do have access to templates. Compared to mobile apps, web apps are usually quicker and easier to build — but they are much simpler in terms of features.',
    ]
)
# [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...]

📚 詳細文檔

模型詳情

模型資源

• 文檔：Sentence Transformers Documentation
• 文檔：Cross Encoder Documentation
• 倉庫：Sentence Transformers on GitHub
• Hugging Face：Cross Encoders on Hugging Face

評估指標

Cross Encoder重排序

• 數據集：gooaq-dev、NanoMSMARCO、NanoNFCorpus和NanoNQ
• 評估方法：使用CrossEncoderRerankingEvaluator進行評估

Cross Encoder Nano BEIR

• 數據集：NanoBEIR_R100_mean
• 評估方法：使用CrossEncoderNanoBEIREvaluator進行評估

訓練詳情

訓練數據集

• 未命名數據集
  • 大小：578,402個訓練樣本
  • 列：question、answer和label
  • 基於前1000個樣本的近似統計信息： | | 問題 | 答案 | 標籤 | | ---- | ---- | ---- | ---- | | 類型 | string | string | int | | 詳情 |
    • 最小長度: 21個字符
    • 平均長度: 43.81個字符
    • 最大長度: 96個字符
    |
    • 最小長度: 51個字符
    • 平均長度: 252.46個字符
    • 最大長度: 405個字符
    |
    • 0: ~82.90%
    • 1: ~17.10%
    |
  • 樣本： | 問題 | 答案 | 標籤 | | ---- | ---- | ---- | | are javascript developers in demand? | JavaScript is the skill that is most in-demand for IT in 2020, according to a report from developer skills tester DevSkiller. The report, “Top IT Skills report 2020: Demand and Hiring Trends,” has JavaScript switching places with Java when compared to last year's report, with Java in third place this year, behind SQL. | 1 | | are javascript developers in demand? | In one line difference between the two is: JavaScript is the programming language where as AngularJS is a framework based on JavaScript. ... It is also the basic for all java script based technologies like jquery, angular JS, bootstrap JS and so on. Angular JS is a framework written in javascript and uses MVC architecture. | 0 | | are javascript developers in demand? | Java applications are run in a virtual machine or web browser while JavaScript is run on a web browser. Java code is compiled whereas while JavaScript code is in text and in a web page. JavaScript is an OOP scripting language, whereas Java is an OOP programming language. | 0 |
• 損失函數：使用BinaryCrossEntropyLoss，參數如下：

{
    "activation_fct": "torch.nn.modules.linear.Identity",
    "pos_weight": 5
}

訓練超參數

非默認超參數

• eval_strategy: steps
• per_device_train_batch_size: 2048
• per_device_eval_batch_size: 2048
• learning_rate: 0.0005
• num_train_epochs: 1
• warmup_ratio: 0.1
• seed: 12
• bf16: True

所有超參數

訓練日誌

環境影響

使用CodeCarbon測量碳排放：

• 能耗：0.019 kWh
• 碳排放：0.007 kg的CO2
• 使用時長：0.099小時

訓練硬件

• 是否使用雲服務：否
• GPU型號：1 x NVIDIA GeForce RTX 3090
• CPU型號：13th Gen Intel(R) Core(TM) i7-13700K
• 內存大小：31.78 GB

框架版本

• Python: 3.11.6
• Sentence Transformers: 3.5.0.dev0
• Transformers: 4.48.3
• PyTorch: 2.5.0+cu121
• Accelerate: 1.3.0
• Datasets: 2.20.0
• Tokenizers: 0.21.0

🔧 技術細節

本模型基於Cross Encoder架構，使用BinaryCrossEntropyLoss作為損失函數進行微調。通過對特定數據集的訓練，模型能夠學習到文本對之間的語義關係，從而為文本對計算得分。在訓練過程中，使用了一系列超參數來控制訓練過程，如學習率、批次大小等。

📄 許可證

本模型使用apache-2.0許可證。

📖 引用

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}