Reranker Bert Tiny Gooaq Bce Tanh V4

🚀 BERT-tiny在GooAQ上训练的模型

这是一个基于Cross Encoder的模型，它使用sentence-transformers库从prajjwal1/bert-tiny微调而来。该模型可以为文本对计算得分，可用于语义文本相似度、语义搜索、释义挖掘、文本分类、聚类等任务。

此模型使用train_script.py进行训练。

🚀 快速开始

本模型是一个经过微调的Cross Encoder模型，可用于计算文本对的得分，适用于多种自然语言处理任务。下面将介绍如何安装依赖库并使用该模型进行推理。

✨ 主要特性

• 多任务适用性：可用于语义文本相似度、语义搜索、释义挖掘、文本分类、聚类等任务。
• 微调基础模型：基于prajjwal1/bert-tiny进行微调，在特定数据集上表现更优。

📦 安装指南

首先，你需要安装sentence-transformers库：

pip install -U sentence-transformers

💻 使用示例

基础用法

from sentence_transformers import CrossEncoder

# 从🤗 Hub下载模型
model = CrossEncoder("cross-encoder-testing/reranker-bert-tiny-gooaq-bce")
# 获取文本对的得分
pairs = [
    ['are javascript developers in demand?', "JavaScript is the skill that is most in-demand for IT in 2020, according to a report from developer skills tester DevSkiller. The report, “Top IT Skills report 2020: Demand and Hiring Trends,” has JavaScript switching places with Java when compared to last year's report, with Java in third place this year, behind SQL."],
    ['are javascript developers in demand?', 'In one line difference between the two is: JavaScript is the programming language where as AngularJS is a framework based on JavaScript. ... It is also the basic for all java script based technologies like jquery, angular JS, bootstrap JS and so on. Angular JS is a framework written in javascript and uses MVC architecture.'],
    ['are javascript developers in demand?', 'Java applications are run in a virtual machine or web browser while JavaScript is run on a web browser. Java code is compiled whereas while JavaScript code is in text and in a web page. JavaScript is an OOP scripting language, whereas Java is an OOP programming language.'],
    ['are javascript developers in demand?', 'Things in the body tag are the things that should be displayed: the actual content. Javascript in the body is executed as it is read and as the page is rendered. Javascript in the head is interpreted before anything is rendered.'],
    ['are javascript developers in demand?', 'Web apps tend to be built using JavaScript, CSS and HTML5. Unlike mobile apps, there is no standard software development kit for building web apps. However, developers do have access to templates. Compared to mobile apps, web apps are usually quicker and easier to build — but they are much simpler in terms of features.'],
]
scores = model.predict(pairs)
print(scores.shape)
# (5,)

# 或者根据与单个文本的相似度对不同文本进行排序
ranks = model.rank(
    'are javascript developers in demand?',
    [
        "JavaScript is the skill that is most in-demand for IT in 2020, according to a report from developer skills tester DevSkiller. The report, “Top IT Skills report 2020: Demand and Hiring Trends,” has JavaScript switching places with Java when compared to last year's report, with Java in third place this year, behind SQL.",
        'In one line difference between the two is: JavaScript is the programming language where as AngularJS is a framework based on JavaScript. ... It is also the basic for all java script based technologies like jquery, angular JS, bootstrap JS and so on. Angular JS is a framework written in javascript and uses MVC architecture.',
        'Java applications are run in a virtual machine or web browser while JavaScript is run on a web browser. Java code is compiled whereas while JavaScript code is in text and in a web page. JavaScript is an OOP scripting language, whereas Java is an OOP programming language.',
        'Things in the body tag are the things that should be displayed: the actual content. Javascript in the body is executed as it is read and as the page is rendered. Javascript in the head is interpreted before anything is rendered.',
        'Web apps tend to be built using JavaScript, CSS and HTML5. Unlike mobile apps, there is no standard software development kit for building web apps. However, developers do have access to templates. Compared to mobile apps, web apps are usually quicker and easier to build — but they are much simpler in terms of features.',
    ]
)
# [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...]

📚 详细文档

模型详情

模型资源

• 文档：Sentence Transformers文档
• 文档：Cross Encoder文档
• 仓库：GitHub上的Sentence Transformers
• Hugging Face：Hugging Face上的Cross Encoders

评估指标

Cross Encoder重排序

• 数据集：gooaq-dev、NanoMSMARCO、NanoNFCorpus和NanoNQ
• 使用CrossEncoderRerankingEvaluator进行评估

Cross Encoder Nano BEIR

• 数据集：NanoBEIR_R100_mean
• 使用CrossEncoderNanoBEIREvaluator进行评估

训练详情

训练数据集

• 未命名数据集
  • 大小：578,402个训练样本
  • 列：question、answer和label
  • 基于前1000个样本的近似统计信息：

    	问题	答案	标签
    类型	字符串	字符串	整数
    详情	最小长度：21个字符 平均长度：43.81个字符 最大长度：96个字符	最小长度：51个字符 平均长度：252.46个字符 最大长度：405个字符	0：约82.90% 1：约17.10%

  • 样本：

    问题	答案	标签
    are javascript developers in demand?	JavaScript is the skill that is most in-demand for IT in 2020, according to a report from developer skills tester DevSkiller. The report, “Top IT Skills report 2020: Demand and Hiring Trends,” has JavaScript switching places with Java when compared to last year's report, with Java in third place this year, behind SQL.	1
    are javascript developers in demand?	In one line difference between the two is: JavaScript is the programming language where as AngularJS is a framework based on JavaScript. ... It is also the basic for all java script based technologies like jquery, angular JS, bootstrap JS and so on. Angular JS is a framework written in javascript and uses MVC architecture.	0
    are javascript developers in demand?	Java applications are run in a virtual machine or web browser while JavaScript is run on a web browser. Java code is compiled whereas while JavaScript code is in text and in a web page. JavaScript is an OOP scripting language, whereas Java is an OOP programming language.	0

• 损失函数：BinaryCrossEntropyLoss，参数如下：

  {
      "activation_fct": "torch.nn.modules.linear.Identity",
      "pos_weight": 5
  }
  

训练超参数

非默认超参数

• eval_strategy：按步骤评估
• per_device_train_batch_size：2048
• per_device_eval_batch_size：2048
• learning_rate：0.0005
• num_train_epochs：1
• warmup_ratio：0.1
• seed：12
• bf16：True

所有超参数

训练日志

环境影响

使用CodeCarbon测量碳排放：

• 能源消耗：0.019千瓦时
• 碳排放：0.007千克二氧化碳
• 使用时长：0.099小时

训练硬件

• 是否使用云服务：否
• GPU型号：1 x NVIDIA GeForce RTX 3090
• CPU型号：13th Gen Intel(R) Core(TM) i7-13700K
• 内存大小：31.78 GB

框架版本

• Python：3.11.6
• Sentence Transformers：3.5.0.dev0
• Transformers：4.48.3
• PyTorch：2.5.0+cu121
• Accelerate：1.3.0
• Datasets：2.20.0
• Tokenizers：0.21.0

📄 许可证

本模型使用apache-2.0许可证。

📚 引用

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}