Open-source BERT Question-answering Model - Suitable for Resource-constrained Environments, Fine-tuned on SQuAD2.0 for Precise Question Answering

Bert Medium Finetuned Squadv2

Developed by mrm8488

A Q&A model fine-tuned on the SQuAD2.0 dataset based on the BERT-Medium architecture, designed for resource-constrained environments

Question Answering System English#Q&A System #Lightweight BERT #SQuAD2.0 Fine-tuning

Downloads 1,399

Release Time : 3/2/2022

Model Overview

This model is part of the small BERT series developed by Google Research, optimized for Q&A tasks, capable of determining whether a passage contains an answer and responding accordingly

Model Features

Lightweight Design

Optimized for resource-constrained environments, with a model size of only 157MB

Dual Q&A Mode

Not only answers questions but also determines whether a passage supports an answer

Efficient Fine-tuning

Fine-tuned on the SQuAD2.0 dataset, balancing model size and performance

Model Capabilities

Text Understanding

Question Answering

No-Answer Judgment

Use Cases

Smart Customer Service

FAQ Auto-Response

Automatically answers common user questions based on knowledge base content

Accuracy 65.95% (EM)

Educational Assistance

Reading Comprehension Aid

Helps students quickly locate key information in articles

F1 Score 70.11%

🚀 BERT-Medium fine-tuned on SQuAD v2

This is a BERT-Medium model created by Google Research. It has been fine-tuned on SQuAD 2.0 for the Q&A downstream task.

Model size (after training): 157.46 MB

✨ Features

Details of BERT-Small and its 'family'

Released on March 11th, 2020, this model is part of 24 smaller BERT models (English only, uncased, trained with WordPiece masking) referenced in Well-Read Students Learn Better: On the Importance of Pre-training Compact Models. These smaller BERT models are designed for environments with restricted computational resources. They can be fine-tuned in the same way as the original BERT models, and are most effective in the context of knowledge distillation, where the fine-tuning labels are produced by a larger and more accurate teacher.

Details of the downstream task (Q&A) - Dataset

SQuAD2.0 combines the 100,000 questions in SQuAD1.1 with over 50,000 unanswerable questions written adversarially by crowdworkers to look similar to answerable ones. To perform well on SQuAD2.0, systems must not only answer questions when possible, but also determine when no answer is supported by the paragraph and abstain from answering.

Dataset	Split	# samples
SQuAD2.0	train	130k
SQuAD2.0	eval	12.3k

📦 Installation

The script for fine tuning can be found here

🔧 Technical Details

Model training

The model was trained on a Tesla P100 GPU and 25GB of RAM.

Results

Metric	# Value
EM	65.95
F1	70.11

Raw metrics from benchmark included in training script

{
  "exact": 65.95637159942727,
  "f1": 70.11632254245896,
  "total": 11873,
  "HasAns_exact": 67.79689608636977,
  "HasAns_f1": 76.12872765631123,
  "HasAns_total": 5928,
  "NoAns_exact": 64.12111017661901,
  "NoAns_f1": 64.12111017661901,
  "NoAns_total": 5945,
  "best_exact": 65.96479407058031,
  "best_exact_thresh": 0.0,
  "best_f1": 70.12474501361196,
  "best_f1_thresh": 0.0
}

Comparison

Model	EM	F1 score	SIZE (MB)
bert-tiny-finetuned-squadv2	48.60	49.73	16.74
bert-tiny-5-finetuned-squadv2	57.12	60.86	24.34
bert-mini-finetuned-squadv2	56.31	59.65	42.63
bert-mini-5-finetuned-squadv2	63.51	66.78	66.76
bert-small-finetuned-squadv2	60.49	64.21	109.74
bert-medium-finetuned-squadv2	65.95	70.11	157.46

💻 Usage Examples

Basic Usage

from transformers import pipeline

qa_pipeline = pipeline(
    "question-answering",
    model="mrm8488/bert-small-finetuned-squadv2",
    tokenizer="mrm8488/bert-small-finetuned-squadv2"
)

qa_pipeline({
    'context': "Manuel Romero has been working hardly in the repository hugginface/transformers lately",
    'question': "Who has been working hard for hugginface/transformers lately?"

})

# Output:

{
  "answer": "Manuel Romero",
  "end": 13,
  "score": 0.9939319924374637,
  "start": 0
}

Advanced Usage

qa_pipeline({
    'context': "Manuel Romero has been working remotely in the repository hugginface/transformers lately",
    'question': "How has been working Manuel Romero?"
})

# Output:

{ "answer": "remotely", "end": 39, "score": 0.3612058272768017, "start": 31 }

Created by Manuel Romero/@mrm8488 | LinkedIn

Made with ♥ in Spain

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご