xtremedistil-l12-h384-uncased Open Source Model - Support Transfer Learning for Arbitrary Tasks and Languages

Xtremedistil L12 H384 Uncased

Developed by microsoft

XtremeDistilTransformers is a task-agnostic Transformer model distilled through task transfer learning, creating a small universal model applicable to any task and language.

Large Language Model

Transformers

EnglishOpen Source License:MIT #Multi-task distillation #Lightweight Transformer #Multilingual support

Downloads 471

Release Time : 3/2/2022

Model Overview

This model combines multi-task distillation techniques to achieve a small universal model through task transfer learning, suitable for various natural language processing tasks and multilingual scenarios.

Model Features

Efficient distillation

Extracts knowledge from large models through multi-stage distillation techniques, significantly reducing model size while maintaining performance.

Task-agnostic

Designed as a task-agnostic model that can be flexibly applied to various natural language processing tasks.

Multilingual support

The model design supports multilingual application scenarios.

Efficient inference

5.3 times faster than BERT-base while maintaining high performance.

Model Capabilities

Text classification

Natural language understanding

Question answering systems

Use Cases

Natural language processing

Text classification

Can be used for various text classification tasks such as sentiment analysis, topic classification, etc.

Excellent performance on GLUE benchmark

Question answering system

Suitable for building question answering systems that can understand questions and provide accurate answers.

Achieved 76.6 F1 score on SQuAD-v2 dataset

🚀 XtremeDistilTransformers for Distilling Massive Neural Networks

XtremeDistilTransformers is a distilled task-agnostic transformer model. It uses task transfer to learn a small universal model that can be applied to arbitrary tasks and languages, as described in the paper XtremeDistilTransformers: Task Transfer for Task-agnostic Distillation.

✨ Features

We combine task transfer with multi-task distillation techniques from the papers XtremeDistil: Multi-stage Distillation for Massive Multilingual Models and MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers. The corresponding Github code is also available.
The l6 - h384 checkpoint has 6 layers, a 384 hidden size, 12 attention heads, and corresponds to 22 million parameters, achieving a 5.3x speedup over BERT - base.
Other available checkpoints: xtremedistil - l6 - h256 - uncased and xtremedistil - l6 - h384 - uncased.

📚 Documentation

The following table shows the results on the GLUE dev set and SQuAD - v2:

Models	#Params	Speedup	MNLI	QNLI	QQP	RTE	SST	MRPC	SQUAD2	Avg
BERT	109	1x	84.5	91.7	91.3	68.6	93.2	87.3	76.8	84.8
DistilBERT	66	2x	82.2	89.2	88.5	59.9	91.3	87.5	70.7	81.3
TinyBERT	66	2x	83.5	90.5	90.6	72.2	91.6	88.4	73.1	84.3
MiniLM	66	2x	84.0	91.0	91.0	71.5	92.0	88.4	76.4	84.9
MiniLM	22	5.3x	82.8	90.3	90.6	68.9	91.3	86.6	72.9	83.3
XtremeDistil - l6 - h256	13	8.7x	83.9	89.5	90.6	80.1	91.2	90.0	74.1	85.6
XtremeDistil - l6 - h384	22	5.3x	85.4	90.3	91.0	80.9	92.3	90.0	76.6	86.6
XtremeDistil - l12 - h384	33	2.7x	87.2	91.9	91.3	85.6	93.1	90.4	80.2	88.5

📄 License

This project is licensed under the MIT license.

📖 Citation

If you use this checkpoint in your work, please cite:

@misc{mukherjee2021xtremedistiltransformers,
      title={XtremeDistilTransformers: Task Transfer for Task-agnostic Distillation}, 
      author={Subhabrata Mukherjee and Ahmed Hassan Awadallah and Jianfeng Gao},
      year={2021},
      eprint={2106.04563},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

⚠️ Important Note

This model was tested with tensorflow 2.3.1, transformers 4.1.1, torch 1.6.0.

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご