Open-source Q&A System of bert-base-uncased-squad1.1-block-sparse-0.32-v1

Bert Base Uncased Squad1.1 Block Sparse 0.32 V1

Developed by madlag

This is a question-answering model based on the BERT-base uncased model fine-tuned on the SQuAD1.1 dataset, employing dynamic pruning techniques to achieve block sparsity while retaining 47% of the original weights.

Question Answering System

Transformers

EnglishOpen Source License:MIT #Question Answering System #Block Sparse Model #SQuAD Fine-tuning

Downloads 21

Release Time : 3/2/2022

Model Overview

This model is a BERT variant optimized for question-answering tasks, reducing model size and improving inference speed through pruning techniques while maintaining high accuracy.

Model Features

Block Sparse Technology

Utilizes dynamic pruning methods, retaining only 31.7% of the original weights in linear layers and 47% overall, improving inference speed.

Attention Head Optimization

80 out of 144 attention heads (55.6%) were removed, optimizing computational efficiency.

Performance Optimization

Evaluation speed is 1.12x faster than dense networks while maintaining high accuracy.

Model Capabilities

Question Answering System

Text Understanding

Context-aware Responses

Use Cases

Question Answering System

Fact-based Question Answering

Answer specific questions based on provided context

EM:79.04, F1:86.70

Educational Applications

Used for automated question-answering systems in the education field

🚀 BERT-base uncased model fine-tuned on SQuAD v1

This project presents a fine - tuned BERT - base uncased model on SQuAD v1. It uses block - sparse techniques to optimize performance, offering a balance between speed and accuracy.

✨ Features

Block Sparse Structure: The linear layers contain 31.7% of the original weights, and the model contains 47.0% of the original weights overall.
Faster Runtime: With the block - sparse runtime, it runs 1.12x faster than dense networks on evaluation, though at the cost of some accuracy.
Case - Insensitive: The model does not distinguish between different cases of English words.
Fine - Tuned and Distilled: Fine - tuned from the HuggingFace BERT base uncased checkpoint on SQuAD1.1 and distilled from [csarron/bert - base - uncased - squad - v1](https://huggingface.co/csarron/bert - base - uncased - squad - v1).

📦 Installation

No specific installation steps are provided in the original README. However, to use this model, you'll need to install the transformers library:

pip install transformers

💻 Usage Examples

Basic Usage

from transformers import pipeline

qa_pipeline = pipeline(
    "question-answering",
    model="madlag/bert-base-uncased-squad1.1-block-sparse-0.32-v1",
    tokenizer="madlag/bert-base-uncased-squad1.1-block-sparse-0.32-v1"
)

predictions = qa_pipeline({
    'context': "Frédéric François Chopin, born Fryderyk Franciszek Chopin (1 March 1810 – 17 October 1849), was a Polish composer and virtuoso pianist of the Romantic era who wrote primarily for solo piano.",
    'question': "Who is Frederic Chopin?",
})

print(predictions)

📚 Documentation

Pruning details

A side - effect of the block pruning is that some of the attention heads are completely removed: 80 heads were removed out of a total of 144 (55.6%).

Here is a detailed view on how the remaining heads are distributed in the network after pruning.

Pruning details

Density plot

Details

Property	Details
Dataset	SQuAD1.1
Train Samples	90.6K
Eval Samples	11.1k

Fine - tuning

Python Version: 3.8.5
Machine Specs:

Memory: 64 GiB
GPUs: 1 GeForce GTX 3090, with 24GiB memory
GPU driver: 455.23.05, CUDA: 11.1

Results

Property	Value	Original (Table 2)
EM	79.04	80.8
F1	86.70	88.5
Pytorch model file size	`355M` (original BERT: `438M`)	-

📄 License

This model is released under the MIT license.

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご