Bert Base Uncased Squadv1.1 Sparse 80 1x4 Block Pruneofa

🚀 80% 1x4 Block Sparse BERT-Base (uncased) Fine Tuned on SQuADv1.1

This model is fine-tuned for question answering on the SQuAD 1.1 dataset, combining pruning and knowledge distillation.

🚀 Quick Start

This model has been fine-tuned for the NLP task of question answering, trained on the SQuAD 1.1 dataset. It is a result of fine-tuning a Prune Once For All 80% 1x4 block sparse pre-trained BERT-Base model, combined with knowledge distillation.

We present a new method for training sparse pre-trained Transformer language models by integrating weight pruning and model distillation. These sparse pre-trained models can be used to transfer learning for a wide range of tasks while maintaining their sparsity pattern. We show how the compressed sparse pre-trained models we trained transfer their knowledge to five different downstream natural language tasks with minimal accuracy loss.

✨ Features

• Fine-tuned for QA: Specifically optimized for the question answering task on the SQuAD 1.1 dataset.
• Sparse Model: Based on a Prune Once For All 80% 1x4 block sparse pre-trained BERT-Base model, combined with knowledge distillation.
• Transfer Learning: Can be used for transfer learning on a variety of tasks while maintaining sparsity.

📦 Installation

No specific installation steps are provided in the original README.

💻 Usage Examples

Basic Usage

Here is how to import this model in Python:

import transformers
import model_compression_research as model_comp

model = transformers.AutoModelForQuestionAnswering.from_pretrained('Intel/bert-base-uncased-squadv1.1-sparse-80-1x4-block-pruneofa')

scheduler = mcr.pruning_scheduler_factory(model, '../../examples/transformers/question-answering/config/lock_config.json')

# Train your model...

scheduler.remove_pruning()

For more code examples, refer to the GitHub Repo.

📚 Documentation

Model Details

Visualization of Prunce Once for All method from Zafrir et al. (2021). More details can be found in their paper.

Intended Use

Metrics (Model Performance)

All the results are the mean of two seperate experiments with the same hyper-parameters and different seeds.

Training and Evaluation Data

Ethical Considerations

Caveats and Recommendations

⚠️ Important Note

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. There are no additional caveats or recommendations for this model.

🔧 Technical Details

The method consists of two steps, teacher preparation and student pruning. The sparse pre-trained model is used for transfer learning while maintaining its sparsity pattern. The method is called Prune Once for All since the pre-trained model is pruned only once while fine-tuning for several language tasks. (Zafrir et al., 2021)

📄 License

This model is licensed under the Apache 2.0 license.

BibTeX entry and citation info

@article{zafrir2021prune,
  title={Prune Once for All: Sparse Pre-Trained Language Models},
  author={Zafrir, Ofir and Larey, Ariel and Boudoukh, Guy and Shen, Haihao and Wasserblat, Moshe},
  journal={arXiv preprint arXiv:2111.05754},
  year={2021}
}