Reranker Bert Tiny Gooaq Bce

🚀 BERT-tiny在GooAQ上训练的模型

这是一个基于Cross Encoder的模型，它使用sentence-transformers库从prajjwal1/bert-tiny微调而来。该模型可以为文本对计算得分，可用于语义文本相似度、语义搜索、释义挖掘、文本分类、聚类等任务。

此模型使用train_script.py进行训练。

🚀 快速开始

本模型是一个基于Cross Encoder的微调模型，可用于计算文本对的得分，适用于语义文本相似度、语义搜索等多种任务。你可以按照以下步骤使用该模型。

✨ 主要特性

• 跨编码器模型：基于Cross Encoder架构，能够有效计算文本对的得分。
• 微调自预训练模型：从prajjwal1/bert-tiny微调而来，结合了预训练模型的优势。
• 多任务适用性：可用于语义文本相似度、语义搜索、释义挖掘、文本分类、聚类等多种任务。

📦 安装指南

首先，你需要安装Sentence Transformers库：

pip install -U sentence-transformers

💻 使用示例

基础用法

以下是一个使用该模型进行文本对得分计算的示例：

from sentence_transformers import CrossEncoder

# 从🤗 Hub下载模型
model = CrossEncoder("cross-encoder-testing/reranker-bert-tiny-gooaq-bce")
# 定义文本对
pairs = [
    ['are javascript developers in demand?', "JavaScript is the skill that is most in-demand for IT in 2020, according to a report from developer skills tester DevSkiller. The report, “Top IT Skills report 2020: Demand and Hiring Trends,” has JavaScript switching places with Java when compared to last year's report, with Java in third place this year, behind SQL."],
    ['are javascript developers in demand?', 'In one line difference between the two is: JavaScript is the programming language where as AngularJS is a framework based on JavaScript. ... It is also the basic for all java script based technologies like jquery, angular JS, bootstrap JS and so on. Angular JS is a framework written in javascript and uses MVC architecture.'],
    ['are javascript developers in demand?', 'Java applications are run in a virtual machine or web browser while JavaScript is run on a web browser. Java code is compiled whereas while JavaScript code is in text and in a web page. JavaScript is an OOP scripting language, whereas Java is an OOP programming language.'],
    ['are javascript developers in demand?', 'Things in the body tag are the things that should be displayed: the actual content. Javascript in the body is executed as it is read and as the page is rendered. Javascript in the head is interpreted before anything is rendered.'],
    ['are javascript developers in demand?', 'Web apps tend to be built using JavaScript, CSS and HTML5. Unlike mobile apps, there is no standard software development kit for building web apps. However, developers do have access to templates. Compared to mobile apps, web apps are usually quicker and easier to build — but they are much simpler in terms of features.'],
]
# 预测得分
scores = model.predict(pairs)
print(scores.shape)
# (5,)

# 或者根据与单个文本的相似度对不同文本进行排序
ranks = model.rank(
    'are javascript developers in demand?',
    [
        "JavaScript is the skill that is most in-demand for IT in 2020, according to a report from developer skills tester DevSkiller. The report, “Top IT Skills report 2020: Demand and Hiring Trends,” has JavaScript switching places with Java when compared to last year's report, with Java in third place this year, behind SQL.",
        'In one line difference between the two is: JavaScript is the programming language where as AngularJS is a framework based on JavaScript. ... It is also the basic for all java script based technologies like jquery, angular JS, bootstrap JS and so on. Angular JS is a framework written in javascript and uses MVC architecture.',
        'Java applications are run in a virtual machine or web browser while JavaScript is run on a web browser. Java code is compiled whereas while JavaScript code is in text and in a web page. JavaScript is an OOP scripting language, whereas Java is an OOP programming language.',
        'Things in the body tag are the things that should be displayed: the actual content. Javascript in the body is executed as it is read and as the page is rendered. Javascript in the head is interpreted before anything is rendered.',
        'Web apps tend to be built using JavaScript, CSS and HTML5. Unlike mobile apps, there is no standard software development kit for building web apps. However, developers do have access to templates. Compared to mobile apps, web apps are usually quicker and easier to build — but they are much simpler in terms of features.',
    ]
)
# [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...]

📚 详细文档

模型详情

模型资源

• 文档：Sentence Transformers文档
• 文档：Cross Encoder文档
• 仓库：GitHub上的Sentence Transformers
• Hugging Face：Hugging Face上的Cross Encoders

评估指标

Cross Encoder重排序

• 数据集：gooaq-dev、NanoMSMARCO、NanoNFCorpus 和 NanoNQ
• 使用CrossEncoderRerankingEvaluator进行评估

Cross Encoder Nano BEIR

• 数据集：NanoBEIR_R100_mean
• 使用CrossEncoderNanoBEIREvaluator进行评估

训练详情

训练数据集

• 未命名数据集
  • 大小：578,402个训练样本
  • 列：question、answer 和 label
  • 基于前1000个样本的近似统计信息：

    	问题	答案	标签
    类型	字符串	字符串	整数
    详情	最小：21个字符 平均：43.81个字符 最大：96个字符	最小：51个字符 平均：252.46个字符 最大：405个字符	0：~82.90% 1：~17.10%

  • 样本：

    问题	答案	标签
    are javascript developers in demand?	JavaScript is the skill that is most in-demand for IT in 2020, according to a report from developer skills tester DevSkiller. The report, “Top IT Skills report 2020: Demand and Hiring Trends,” has JavaScript switching places with Java when compared to last year's report, with Java in third place this year, behind SQL.	1
    are javascript developers in demand?	In one line difference between the two is: JavaScript is the programming language where as AngularJS is a framework based on JavaScript. ... It is also the basic for all java script based technologies like jquery, angular JS, bootstrap JS and so on. Angular JS is a framework written in javascript and uses MVC architecture.	0
    are javascript developers in demand?	Java applications are run in a virtual machine or web browser while JavaScript is run on a web browser. Java code is compiled whereas while JavaScript code is in text and in a web page. JavaScript is an OOP scripting language, whereas Java is an OOP programming language.	0

• 损失函数：BinaryCrossEntropyLoss，参数如下：

  {
      "activation_fct": "torch.nn.modules.linear.Identity",
      "pos_weight": 5
  }
  

训练超参数

• 非默认超参数
  • eval_strategy: steps
  • per_device_train_batch_size: 2048
  • per_device_eval_batch_size: 2048
  • learning_rate: 0.0005
  • num_train_epochs: 1
  • warmup_ratio: 0.1
  • seed: 12
  • bf16: True
• 所有超参数：点击下面的展开按钮查看

训练日志

环境影响

使用CodeCarbon测量碳排放：

• 能源消耗：0.019 kWh
• 碳排放：0.007 kg CO2
• 使用时长：0.099小时

训练硬件

• 是否使用云服务：否
• GPU型号：1 x NVIDIA GeForce RTX 3090
• CPU型号：13th Gen Intel(R) Core(TM) i7-13700K
• 内存大小：31.78 GB

框架版本

• Python: 3.11.6
• Sentence Transformers: 3.5.0.dev0
• Transformers: 4.48.3
• PyTorch: 2.5.0+cu121
• Accelerate: 1.3.0
• Datasets: 2.20.0
• Tokenizers: 0.21.0

📄 许可证

本项目采用apache-2.0许可证。

📖 引用

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}