xtremedistil-l6-h256-uncased Open-source Model - Supports Multi-task and Multi-language, Freely Meets Diverse Needs

Xtremedistil L6 H256 Uncased

Developed by microsoft

XtremeDistilTransformers is a task-agnostic distilled Transformer model that utilizes task transfer learning techniques to train small general-purpose models suitable for various tasks and languages.

Large Language Model

Transformers

EnglishOpen Source License:MIT #Multi-task Distillation #Multilingual Support #Efficient Inference

Downloads 3,816

Release Time : 3/2/2022

Model Overview

This model combines multi-task distillation techniques, featuring a 6-layer network structure with 384-dimensional hidden layers and 22 million parameters, achieving 5.3x speedup compared to BERT-base.

Model Features

Task-Agnostic Distillation

Trained using task transfer learning techniques, applicable to any task and language.

Efficient Compression

Achieves 5.3x speedup compared to BERT-base with 80% fewer parameters.

Multi-Task Distillation

Incorporates advanced distillation methods from both XtremeDistil and MiniLM papers.

High Performance

Performs excellently on benchmarks like GLUE and SQuAD-v2, approaching the performance of the original large models.

Model Capabilities

Text Classification

Question Answering Systems

Natural Language Understanding

Semantic Similarity Calculation

Use Cases

Natural Language Processing

Text Classification

Can be used for sentiment analysis, topic classification, and other tasks.

Achieves 92.3% accuracy on SST-2 sentiment analysis task.

Question Answering

Suitable for open-domain question answering tasks.

Achieves 76.6 F1 score on SQuAD-v2 question answering task.

Semantic Similarity

Can be used to determine the semantic similarity between two texts.

Achieves 91.0% accuracy on QQP semantic similarity task.

🚀 XtremeDistilTransformers for Distilling Massive Neural Networks

XtremeDistilTransformers is a distilled task-agnostic transformer model. It uses task transfer to learn a small universal model that can be applied to arbitrary tasks and languages, as described in the paper XtremeDistilTransformers: Task Transfer for Task-agnostic Distillation.

✨ Features

We combine task transfer with multi-task distillation techniques from the papers XtremeDistil: Multi-stage Distillation for Massive Multilingual Models and MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers. The related Github code is also available.
This l6 - h384 checkpoint has 6 layers, a 384 hidden size, 12 attention heads, and corresponds to 22 million parameters, achieving a 5.3x speedup over BERT - base.
Other available checkpoints: xtremedistil-l6-h384-uncased and xtremedistil-l12-h384-uncased.

📚 Documentation

The following table shows the results on GLUE dev set and SQuAD - v2.

Models	#Params	Speedup	MNLI	QNLI	QQP	RTE	SST	MRPC	SQUAD2	Avg
BERT	109	1x	84.5	91.7	91.3	68.6	93.2	87.3	76.8	84.8
DistilBERT	66	2x	82.2	89.2	88.5	59.9	91.3	87.5	70.7	81.3
TinyBERT	66	2x	83.5	90.5	90.6	72.2	91.6	88.4	73.1	84.3
MiniLM	66	2x	84.0	91.0	91.0	71.5	92.0	88.4	76.4	84.9
MiniLM	22	5.3x	82.8	90.3	90.6	68.9	91.3	86.6	72.9	83.3
XtremeDistil-l6-h256	13	8.7x	83.9	89.5	90.6	80.1	91.2	90.0	74.1	85.6
XtremeDistil-l6-h384	22	5.3x	85.4	90.3	91.0	80.9	92.3	90.0	76.6	86.6
XtremeDistil-l12-h384	33	2.7x	87.2	91.9	91.3	85.6	93.1	90.4	80.2	88.5

🔧 Technical Details

Tested with tensorflow 2.3.1, transformers 4.1.1, torch 1.6.0

📄 License

This project is under the MIT license.

📖 Citation

If you use this checkpoint in your work, please cite:

@misc{mukherjee2021xtremedistiltransformers,
      title={XtremeDistilTransformers: Task Transfer for Task-agnostic Distillation}, 
      author={Subhabrata Mukherjee and Ahmed Hassan Awadallah and Jianfeng Gao},
      year={2021},
      eprint={2106.04563},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご