Open-source XtremeDistilTransformers Model - Lightweight and Suitable for Various Natural Language Processing Tasks

Xtremedistil L6 H384 Uncased

Developed by microsoft

XtremeDistilTransformers is a knowledge-distilled lightweight Transformer model with task-agnostic properties, applicable to various natural language processing tasks.

Large Language Model EnglishOpen Source License:MIT #Multi-task Distillation #Lightweight Transformer #Multilingual Support

Downloads 1,854

Release Time : 3/2/2022

Model Overview

This model employs task transfer techniques for knowledge distillation, learning a small general-purpose model suitable for any task and language. Compared to BERT-base, it achieves a 5.3x speedup.

Model Features

Task-Agnostic Distillation

Learns a small general-purpose model through task transfer techniques, applicable to any task and language.

Efficient Performance

Achieves a 5.3x speedup compared to BERT-base while maintaining high performance.

Multi-task Distillation Technique

Incorporates multi-task distillation techniques like XtremeDistil and MiniLM to optimize model performance.

Model Capabilities

Text Classification

Question Answering System

Natural Language Understanding

Use Cases

Natural Language Processing

Text Classification

Can be used for sentiment analysis, topic classification, and other text classification tasks.

Achieves 85.4% accuracy on the MNLI task of the GLUE development set.

Question Answering System

Suitable for building efficient question answering systems.

Achieves 76.6% F1 score on the SQuAD-v2 task.

🚀 XtremeDistilTransformers for Distilling Massive Neural Networks

XtremeDistilTransformers is a distilled task-agnostic transformer model. It uses task transfer to learn a small universal model applicable to arbitrary tasks and languages, as described in the paper XtremeDistilTransformers: Task Transfer for Task-agnostic Distillation.

We combine task transfer with multi-task distillation techniques from the papers XtremeDistil: Multi-stage Distillation for Massive Multilingual Models and MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers, and you can find the related Github code.

📚 Documentation

This l6 - h384 checkpoint has 6 layers, a 384 hidden size, 12 attention heads, and corresponds to 22 million parameters. It offers a 5.3x speedup over BERT - base.

Other available checkpoints are xtremedistil - l6 - h256 - uncased and xtremedistil - l12 - h384 - uncased.

The following table presents the results on the GLUE dev set and SQuAD - v2:

Property	Details
Model Type	XtremeDistilTransformers, a distilled task - agnostic transformer model
Training Data	Not specified in the original text

Models	#Params	Speedup	MNLI	QNLI	QQP	RTE	SST	MRPC	SQUAD2	Avg
BERT	109	1x	84.5	91.7	91.3	68.6	93.2	87.3	76.8	84.8
DistilBERT	66	2x	82.2	89.2	88.5	59.9	91.3	87.5	70.7	81.3
TinyBERT	66	2x	83.5	90.5	90.6	72.2	91.6	88.4	73.1	84.3
MiniLM	66	2x	84.0	91.0	91.0	71.5	92.0	88.4	76.4	84.9
MiniLM	22	5.3x	82.8	90.3	90.6	68.9	91.3	86.6	72.9	83.3
XtremeDistil - l6 - h256	13	8.7x	83.9	89.5	90.6	80.1	91.2	90.0	74.1	85.6
XtremeDistil - l6 - h384	22	5.3x	85.4	90.3	91.0	80.9	92.3	90.0	76.6	86.6
XtremeDistil - l12 - h384	33	2.7x	87.2	91.9	91.3	85.6	93.1	90.4	80.2	88.5

🔧 Technical Details

The model was tested with tensorflow 2.3.1, transformers 4.1.1, torch 1.6.0.

📄 License

This project is released under the MIT license.

📖 Citation

If you use this checkpoint in your work, please cite:

@misc{mukherjee2021xtremedistiltransformers,
      title={XtremeDistilTransformers: Task Transfer for Task-agnostic Distillation}, 
      author={Subhabrata Mukherjee and Ahmed Hassan Awadallah and Jianfeng Gao},
      year={2021},
      eprint={2106.04563},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご