Distilbart Mnli 12 1

🚀 DistilBart-MNLI

DistilBart-MNLI is a distilled version of bart-large-mnli, which offers a more lightweight alternative while maintaining high performance in zero - shot classification tasks.

✨ Features

• Distillation Technique: DistilBart-MNLI is created using the No Teacher Distillation technique proposed by Huggingface for BART summarisation, as described here. It simply copies alternating layers from bart-large-mnli and fine - tunes on the same data.
• Performance: Despite being a distilled model, it shows only a slight performance drop compared to the baseline bart-large-mnli.

📦 Installation

If you want to train these models yourself, you need to clone the distillbart-mnli repo and follow the steps below:

Clone and install transformers from source

git clone https://github.com/huggingface/transformers.git
pip install -qqq -U ./transformers

Download MNLI data

python transformers/utils/download_glue_data.py --data_dir glue_data --tasks MNLI

Create student model

python create_student.py \
  --teacher_model_name_or_path facebook/bart-large-mnli \
  --student_encoder_layers 12 \
  --student_decoder_layers 6 \
  --save_path student-bart-mnli-12-6 \

Start fine - tuning

python run_glue.py args.json

📚 Documentation

Performance Comparison

This shows that the distillation technique is very simple and effective, with only a minor performance drop.

Detailed Performance Trade - offs

Detailed performance trade - offs will be posted in this sheet.

Training Logs

You can find the logs of these trained models in this wandb project.