Distilbart Mnli 12 3

🚀 DistilBart-MNLI

DistilBart-MNLI is a distilled version of bart-large-mnli. It is created using the No Teacher Distillation technique proposed by Huggingface for BART summarisation, which can be found here.

The model is developed by copying alternating layers from bart-large-mnli and then fine - tuning on the same dataset.

✨ Features

• Datasets:
  • mnli
• Tags:
  • distilbart
  • distilbart-mnli
• Pipeline Tag: zero - shot - classification

📊 Performance Comparison

This simple yet effective technique results in only a minor drop in performance. Detailed performance trade - offs can be found in this sheet.

📦 Installation and Fine - tuning

If you wish to train these models on your own, follow these steps:

1. Clone and install transformers from source

git clone https://github.com/huggingface/transformers.git
pip install -qqq -U ./transformers

2. Download MNLI data

python transformers/utils/download_glue_data.py --data_dir glue_data --tasks MNLI

3. Create student model

python create_student.py \
  --teacher_model_name_or_path facebook/bart-large-mnli \
  --student_encoder_layers 12 \
  --student_decoder_layers 6 \
  --save_path student-bart-mnli-12-6 \

4. Start fine - tuning

python run_glue.py args.json

You can access the logs of these trained models in this wandb project.