Reranker Bert Tiny Gooaq Bce Tanh V4

🚀 BERT-tiny在GooAQ上訓練的模型

這是一個基於Cross Encoder的模型，它使用sentence-transformers庫從prajjwal1/bert-tiny微調而來。該模型可以為文本對計算得分，可用於語義文本相似度、語義搜索、釋義挖掘、文本分類、聚類等任務。

此模型使用train_script.py進行訓練。

🚀 快速開始

本模型是一個經過微調的Cross Encoder模型，可用於計算文本對的得分，適用於多種自然語言處理任務。下面將介紹如何安裝依賴庫並使用該模型進行推理。

✨ 主要特性

• 多任務適用性：可用於語義文本相似度、語義搜索、釋義挖掘、文本分類、聚類等任務。
• 微調基礎模型：基於prajjwal1/bert-tiny進行微調，在特定數據集上表現更優。

📦 安裝指南

首先，你需要安裝sentence-transformers庫：

pip install -U sentence-transformers

💻 使用示例

基礎用法

from sentence_transformers import CrossEncoder

# 從🤗 Hub下載模型
model = CrossEncoder("cross-encoder-testing/reranker-bert-tiny-gooaq-bce")
# 獲取文本對的得分
pairs = [
    ['are javascript developers in demand?', "JavaScript is the skill that is most in-demand for IT in 2020, according to a report from developer skills tester DevSkiller. The report, “Top IT Skills report 2020: Demand and Hiring Trends,” has JavaScript switching places with Java when compared to last year's report, with Java in third place this year, behind SQL."],
    ['are javascript developers in demand?', 'In one line difference between the two is: JavaScript is the programming language where as AngularJS is a framework based on JavaScript. ... It is also the basic for all java script based technologies like jquery, angular JS, bootstrap JS and so on. Angular JS is a framework written in javascript and uses MVC architecture.'],
    ['are javascript developers in demand?', 'Java applications are run in a virtual machine or web browser while JavaScript is run on a web browser. Java code is compiled whereas while JavaScript code is in text and in a web page. JavaScript is an OOP scripting language, whereas Java is an OOP programming language.'],
    ['are javascript developers in demand?', 'Things in the body tag are the things that should be displayed: the actual content. Javascript in the body is executed as it is read and as the page is rendered. Javascript in the head is interpreted before anything is rendered.'],
    ['are javascript developers in demand?', 'Web apps tend to be built using JavaScript, CSS and HTML5. Unlike mobile apps, there is no standard software development kit for building web apps. However, developers do have access to templates. Compared to mobile apps, web apps are usually quicker and easier to build — but they are much simpler in terms of features.'],
]
scores = model.predict(pairs)
print(scores.shape)
# (5,)

# 或者根據與單個文本的相似度對不同文本進行排序
ranks = model.rank(
    'are javascript developers in demand?',
    [
        "JavaScript is the skill that is most in-demand for IT in 2020, according to a report from developer skills tester DevSkiller. The report, “Top IT Skills report 2020: Demand and Hiring Trends,” has JavaScript switching places with Java when compared to last year's report, with Java in third place this year, behind SQL.",
        'In one line difference between the two is: JavaScript is the programming language where as AngularJS is a framework based on JavaScript. ... It is also the basic for all java script based technologies like jquery, angular JS, bootstrap JS and so on. Angular JS is a framework written in javascript and uses MVC architecture.',
        'Java applications are run in a virtual machine or web browser while JavaScript is run on a web browser. Java code is compiled whereas while JavaScript code is in text and in a web page. JavaScript is an OOP scripting language, whereas Java is an OOP programming language.',
        'Things in the body tag are the things that should be displayed: the actual content. Javascript in the body is executed as it is read and as the page is rendered. Javascript in the head is interpreted before anything is rendered.',
        'Web apps tend to be built using JavaScript, CSS and HTML5. Unlike mobile apps, there is no standard software development kit for building web apps. However, developers do have access to templates. Compared to mobile apps, web apps are usually quicker and easier to build — but they are much simpler in terms of features.',
    ]
)
# [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...]

📚 詳細文檔

模型詳情

模型資源

• 文檔：Sentence Transformers文檔
• 文檔：Cross Encoder文檔
• 倉庫：GitHub上的Sentence Transformers
• Hugging Face：Hugging Face上的Cross Encoders

評估指標

Cross Encoder重排序

• 數據集：gooaq-dev、NanoMSMARCO、NanoNFCorpus和NanoNQ
• 使用CrossEncoderRerankingEvaluator進行評估

Cross Encoder Nano BEIR

• 數據集：NanoBEIR_R100_mean
• 使用CrossEncoderNanoBEIREvaluator進行評估

訓練詳情

訓練數據集

• 未命名數據集
  • 大小：578,402個訓練樣本
  • 列：question、answer和label
  • 基於前1000個樣本的近似統計信息：

    	問題	答案	標籤
    類型	字符串	字符串	整數
    詳情	最小長度：21個字符 平均長度：43.81個字符 最大長度：96個字符	最小長度：51個字符 平均長度：252.46個字符 最大長度：405個字符	0：約82.90% 1：約17.10%

  • 樣本：

    問題	答案	標籤
    are javascript developers in demand?	JavaScript is the skill that is most in-demand for IT in 2020, according to a report from developer skills tester DevSkiller. The report, “Top IT Skills report 2020: Demand and Hiring Trends,” has JavaScript switching places with Java when compared to last year's report, with Java in third place this year, behind SQL.	1
    are javascript developers in demand?	In one line difference between the two is: JavaScript is the programming language where as AngularJS is a framework based on JavaScript. ... It is also the basic for all java script based technologies like jquery, angular JS, bootstrap JS and so on. Angular JS is a framework written in javascript and uses MVC architecture.	0
    are javascript developers in demand?	Java applications are run in a virtual machine or web browser while JavaScript is run on a web browser. Java code is compiled whereas while JavaScript code is in text and in a web page. JavaScript is an OOP scripting language, whereas Java is an OOP programming language.	0

• 損失函數：BinaryCrossEntropyLoss，參數如下：

  {
      "activation_fct": "torch.nn.modules.linear.Identity",
      "pos_weight": 5
  }
  

訓練超參數

非默認超參數

• eval_strategy：按步驟評估
• per_device_train_batch_size：2048
• per_device_eval_batch_size：2048
• learning_rate：0.0005
• num_train_epochs：1
• warmup_ratio：0.1
• seed：12
• bf16：True

所有超參數

訓練日誌

環境影響

使用CodeCarbon測量碳排放：

• 能源消耗：0.019千瓦時
• 碳排放：0.007千克二氧化碳
• 使用時長：0.099小時

訓練硬件

• 是否使用雲服務：否
• GPU型號：1 x NVIDIA GeForce RTX 3090
• CPU型號：13th Gen Intel(R) Core(TM) i7-13700K
• 內存大小：31.78 GB

框架版本

• Python：3.11.6
• Sentence Transformers：3.5.0.dev0
• Transformers：4.48.3
• PyTorch：2.5.0+cu121
• Accelerate：1.3.0
• Datasets：2.20.0
• Tokenizers：0.21.0

📄 許可證

本模型使用apache-2.0許可證。

📚 引用

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}