bert-base-uncased-squad1.1-block-sparse-0.13-v1 Open-source Q&A Model - Precise Answers to Facilitate Information Acquisition

Bert Base Uncased Squad1.1 Block Sparse 0.13 V1

Developed by madlag

This is a question answering system model based on BERT-base-uncased fine-tuned on the SQuAD1.1 dataset, featuring a block sparse structure that retains 32.1% of the original model's weights.

Question Answering System

Transformers

EnglishOpen Source License:MIT #Question Answering System #Block Sparse Structure #Dynamic Pruning

Downloads 25

Release Time : 3/2/2022

Model Overview

The model is primarily used for question answering tasks, capable of answering relevant questions based on provided context. It is case-insensitive and employs dynamic pruning technology to improve evaluation speed.

Model Features

Block Sparse Structure

Linear layers retain only 12.5% of original weights, with overall 32.1% weight retention, achieving 1.65x faster evaluation speed compared to dense networks.

Dynamic Pruning

Utilizes Victor Sanh's improved version of dynamic pruning to optimize model performance.

Attention Head Removal

97 out of 144 attention heads (67.4%) were removed to further optimize the model structure.

Model Capabilities

Question Answering System

Text Understanding

Contextual Answering

Use Cases

Education

Historical Knowledge QA

Answer questions based on historical texts, e.g., 'Where is the Eiffel Tower located?'

Can accurately answer questions within the provided context.

Information Retrieval

Document QA

Extract information from documents and answer related questions.

Can provide accurate answers based on document content.

🚀 BERT-base uncased model fine-tuned on SQuAD v1

This model is a fine-tuned BERT-base uncased model on SQuAD v1, featuring block sparsity for improved runtime performance.

🚀 Quick Start

This model is block sparse: the linear layers contains 12.5% of the original weights. The model contains 32.1% of the original weights overall. The training use a modified version of Victor Sanh Movement Pruning method. That means that with the block-sparse runtime it ran 1.65x faster than an dense networks on the evaluation, at the price of some impact on the accuracy (see below).

This model was fine-tuned from the HuggingFace BERT base uncased checkpoint on SQuAD1.1, and distilled from the equivalent model csarron/bert-base-uncased-squad-v1. This model is case-insensitive: it does not make a difference between english and English.

✨ Features

Block sparse architecture, with linear layers containing 12.5% of the original weights and the overall model containing 32.1% of the original weights.
Trained using a modified Movement Pruning method for faster evaluation runtime.
Fine-tuned on SQuAD1.1 and distilled from an equivalent model.
Case-insensitive.

🔧 Technical Details

Pruning details

A side-effect of the block pruning is that some of the attention heads are completely removed: 97 heads were removed on a total of 144 (67.4%). Here is a detailed view on how the remaining heads are distributed in the network after pruning.

Pruning details

Density plot

Details

Dataset	Split	# samples
SQuAD1.1	train	90.6K
SQuAD1.1	eval	11.1k

Fine-tuning

Python: 3.8.5
Machine specs:

Memory: 64 GiB
GPUs: 1 GeForce GTX 3090, with 24GiB memory
GPU driver: 455.23.05, CUDA: 11.1

Results

Pytorch model file size: 342M (original BERT: 438M)

Metric	# Value	# Original (Table 2)
EM	74.39	80.8
F1	83.26	88.5

💻 Usage Examples

Basic Usage

from transformers import pipeline

qa_pipeline = pipeline(
    "question-answering",
    model="madlag/bert-base-uncased-squad1.1-block-sparse-0.13-v1",
    tokenizer="madlag/bert-base-uncased-squad1.1-block-sparse-0.13-v1"
)

predictions = qa_pipeline({
    'context': "Frédéric François Chopin, born Fryderyk Franciszek Chopin (1 March 1810 – 17 October 1849), was a Polish composer and virtuoso pianist of the Romantic era who wrote primarily for solo piano.",
    'question': "Who is Frederic Chopin?",
})

print(predictions)

📄 License

This model is released under the MIT license.

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご