Reranker Bert Tiny Gooaq Bce

🚀 BERT-tiny在GooAQ上訓練的模型

這是一個基於Cross Encoder的模型，它使用sentence-transformers庫從prajjwal1/bert-tiny微調而來。該模型可以為文本對計算得分，可用於語義文本相似度、語義搜索、釋義挖掘、文本分類、聚類等任務。

此模型使用train_script.py進行訓練。

🚀 快速開始

本模型是一個基於Cross Encoder的微調模型，可用於計算文本對的得分，適用於語義文本相似度、語義搜索等多種任務。你可以按照以下步驟使用該模型。

✨ 主要特性

• 跨編碼器模型：基於Cross Encoder架構，能夠有效計算文本對的得分。
• 微調自預訓練模型：從prajjwal1/bert-tiny微調而來，結合了預訓練模型的優勢。
• 多任務適用性：可用於語義文本相似度、語義搜索、釋義挖掘、文本分類、聚類等多種任務。

📦 安裝指南

首先，你需要安裝Sentence Transformers庫：

pip install -U sentence-transformers

💻 使用示例

基礎用法

以下是一個使用該模型進行文本對得分計算的示例：

from sentence_transformers import CrossEncoder

# 從🤗 Hub下載模型
model = CrossEncoder("cross-encoder-testing/reranker-bert-tiny-gooaq-bce")
# 定義文本對
pairs = [
    ['are javascript developers in demand?', "JavaScript is the skill that is most in-demand for IT in 2020, according to a report from developer skills tester DevSkiller. The report, “Top IT Skills report 2020: Demand and Hiring Trends,” has JavaScript switching places with Java when compared to last year's report, with Java in third place this year, behind SQL."],
    ['are javascript developers in demand?', 'In one line difference between the two is: JavaScript is the programming language where as AngularJS is a framework based on JavaScript. ... It is also the basic for all java script based technologies like jquery, angular JS, bootstrap JS and so on. Angular JS is a framework written in javascript and uses MVC architecture.'],
    ['are javascript developers in demand?', 'Java applications are run in a virtual machine or web browser while JavaScript is run on a web browser. Java code is compiled whereas while JavaScript code is in text and in a web page. JavaScript is an OOP scripting language, whereas Java is an OOP programming language.'],
    ['are javascript developers in demand?', 'Things in the body tag are the things that should be displayed: the actual content. Javascript in the body is executed as it is read and as the page is rendered. Javascript in the head is interpreted before anything is rendered.'],
    ['are javascript developers in demand?', 'Web apps tend to be built using JavaScript, CSS and HTML5. Unlike mobile apps, there is no standard software development kit for building web apps. However, developers do have access to templates. Compared to mobile apps, web apps are usually quicker and easier to build — but they are much simpler in terms of features.'],
]
# 預測得分
scores = model.predict(pairs)
print(scores.shape)
# (5,)

# 或者根據與單個文本的相似度對不同文本進行排序
ranks = model.rank(
    'are javascript developers in demand?',
    [
        "JavaScript is the skill that is most in-demand for IT in 2020, according to a report from developer skills tester DevSkiller. The report, “Top IT Skills report 2020: Demand and Hiring Trends,” has JavaScript switching places with Java when compared to last year's report, with Java in third place this year, behind SQL.",
        'In one line difference between the two is: JavaScript is the programming language where as AngularJS is a framework based on JavaScript. ... It is also the basic for all java script based technologies like jquery, angular JS, bootstrap JS and so on. Angular JS is a framework written in javascript and uses MVC architecture.',
        'Java applications are run in a virtual machine or web browser while JavaScript is run on a web browser. Java code is compiled whereas while JavaScript code is in text and in a web page. JavaScript is an OOP scripting language, whereas Java is an OOP programming language.',
        'Things in the body tag are the things that should be displayed: the actual content. Javascript in the body is executed as it is read and as the page is rendered. Javascript in the head is interpreted before anything is rendered.',
        'Web apps tend to be built using JavaScript, CSS and HTML5. Unlike mobile apps, there is no standard software development kit for building web apps. However, developers do have access to templates. Compared to mobile apps, web apps are usually quicker and easier to build — but they are much simpler in terms of features.',
    ]
)
# [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...]

📚 詳細文檔

模型詳情

模型資源

• 文檔：Sentence Transformers文檔
• 文檔：Cross Encoder文檔
• 倉庫：GitHub上的Sentence Transformers
• Hugging Face：Hugging Face上的Cross Encoders

評估指標

Cross Encoder重排序

• 數據集：gooaq-dev、NanoMSMARCO、NanoNFCorpus 和 NanoNQ
• 使用CrossEncoderRerankingEvaluator進行評估

Cross Encoder Nano BEIR

• 數據集：NanoBEIR_R100_mean
• 使用CrossEncoderNanoBEIREvaluator進行評估

訓練詳情

訓練數據集

• 未命名數據集
  • 大小：578,402個訓練樣本
  • 列：question、answer 和 label
  • 基於前1000個樣本的近似統計信息：

    	問題	答案	標籤
    類型	字符串	字符串	整數
    詳情	最小：21個字符 平均：43.81個字符 最大：96個字符	最小：51個字符 平均：252.46個字符 最大：405個字符	0：~82.90% 1：~17.10%

  • 樣本：

    問題	答案	標籤
    are javascript developers in demand?	JavaScript is the skill that is most in-demand for IT in 2020, according to a report from developer skills tester DevSkiller. The report, “Top IT Skills report 2020: Demand and Hiring Trends,” has JavaScript switching places with Java when compared to last year's report, with Java in third place this year, behind SQL.	1
    are javascript developers in demand?	In one line difference between the two is: JavaScript is the programming language where as AngularJS is a framework based on JavaScript. ... It is also the basic for all java script based technologies like jquery, angular JS, bootstrap JS and so on. Angular JS is a framework written in javascript and uses MVC architecture.	0
    are javascript developers in demand?	Java applications are run in a virtual machine or web browser while JavaScript is run on a web browser. Java code is compiled whereas while JavaScript code is in text and in a web page. JavaScript is an OOP scripting language, whereas Java is an OOP programming language.	0

• 損失函數：BinaryCrossEntropyLoss，參數如下：

  {
      "activation_fct": "torch.nn.modules.linear.Identity",
      "pos_weight": 5
  }
  

訓練超參數

• 非默認超參數
  • eval_strategy: steps
  • per_device_train_batch_size: 2048
  • per_device_eval_batch_size: 2048
  • learning_rate: 0.0005
  • num_train_epochs: 1
  • warmup_ratio: 0.1
  • seed: 12
  • bf16: True
• 所有超參數：點擊下面的展開按鈕查看

訓練日誌

環境影響

使用CodeCarbon測量碳排放：

• 能源消耗：0.019 kWh
• 碳排放：0.007 kg CO2
• 使用時長：0.099小時

訓練硬件

• 是否使用雲服務：否
• GPU型號：1 x NVIDIA GeForce RTX 3090
• CPU型號：13th Gen Intel(R) Core(TM) i7-13700K
• 內存大小：31.78 GB

框架版本

• Python: 3.11.6
• Sentence Transformers: 3.5.0.dev0
• Transformers: 4.48.3
• PyTorch: 2.5.0+cu121
• Accelerate: 1.3.0
• Datasets: 2.20.0
• Tokenizers: 0.21.0

📄 許可證

本項目採用apache-2.0許可證。

📖 引用

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}