Deberta V2 Xxlarge Mnli

🚀 DeBERTa: Decoding-enhanced BERT with Disentangled Attention

DeBERTa improves the BERT and RoBERTa models using disentangled attention and enhanced mask decoder. It outperforms BERT and RoBERTa on the majority of NLU tasks with 80GB training data.

Please check the official repository for more details and updates. This is the DeBERTa V2 XXLarge model fine - tuned with the MNLI task, having 48 layers and a 1536 hidden size. The total number of parameters is 1.5B.

🚀 Quick Start

This section provides an overview of the model and its performance, along with citation information. For more detailed usage, continue reading the following sections.

✨ Features

• Improved Architecture: DeBERTa enhances BERT and RoBERTa using disentangled attention and an enhanced mask decoder.
• High Performance: Outperforms BERT and RoBERTa on most NLU tasks with 80GB of training data.
• Fine - Tuned Model: The DeBERTa V2 XXLarge model is fine - tuned for the MNLI task.

📚 Documentation

Fine - tuning on NLU tasks

We present the dev results on SQuAD 1.1/2.0 and several GLUE benchmark tasks.

Notes.

• ¹ Following RoBERTa, for RTE, MRPC, STS - B, we fine - tune the tasks based on [DeBERTa - Large - MNLI](https://huggingface.co/microsoft/deberta - large - mnli), [DeBERTa - XLarge - MNLI](https://huggingface.co/microsoft/deberta - xlarge - mnli), [DeBERTa - V2 - XLarge - MNLI](https://huggingface.co/microsoft/deberta - v2 - xlarge - mnli), [DeBERTa - V2 - XXLarge - MNLI](https://huggingface.co/microsoft/deberta - v2 - xxlarge - mnli). The results of SST - 2/QQP/QNLI/SQuADv2 will also be slightly improved when starting from MNLI fine - tuned models. However, we only report the numbers fine - tuned from pretrained base models for those 4 tasks.
• ² To try the XXLarge model with HF transformers, we recommend using deepspeed as it's faster and saves memory.

💻 Usage Examples

Basic Usage

Run with Deepspeed:

pip install datasets
pip install deepspeed

# Download the deepspeed config file
wget https://huggingface.co/microsoft/deberta - v2 - xxlarge - mnli/resolve/main/ds_config.json -O ds_config.json

export TASK_NAME=rte
output_dir="ds_results"
num_gpus=8
batch_size=4
python -m torch.distributed.launch --nproc_per_node=${num_gpus} \
  run_glue.py \
  --model_name_or_path microsoft/deberta - v2 - xxlarge - mnli \
  --task_name $TASK_NAME \
  --do_train \
  --do_eval \
  --max_seq_length 256 \
  --per_device_train_batch_size ${batch_size} \
  --learning_rate 3e-6 \
  --num_train_epochs 3 \
  --output_dir $output_dir \
  --overwrite_output_dir \
  --logging_steps 10 \
  --logging_dir $output_dir \
  --deepspeed ds_config.json

Advanced Usage

You can also run with --sharded_ddp

cd transformers/examples/text - classification/
export TASK_NAME=rte
python -m torch.distributed.launch --nproc_per_node=8 run_glue.py   --model_name_or_path microsoft/deberta - v2 - xxlarge - mnli   \
--task_name $TASK_NAME   --do_train   --do_eval   --max_seq_length 256   --per_device_train_batch_size 4   \
--learning_rate 3e-6   --num_train_epochs 3   --output_dir /tmp/$TASK_NAME/ --overwrite_output_dir --sharded_ddp --fp16

📄 License

This project is licensed under the MIT license.

🔧 Technical Details

• Model Architecture: DeBERTa uses disentangled attention and an enhanced mask decoder to improve upon BERT and RoBERTa.
• Fine - Tuning: Fine - tuned on the MNLI task, with specific configurations for different GLUE benchmark tasks.
• Performance: Demonstrates high performance on SQuAD 1.1/2.0 and multiple GLUE tasks compared to other models like BERT - Large, RoBERTa - Large, etc.

📚 Citation

If you find DeBERTa useful for your work, please cite the following paper:

@inproceedings{
he2021deberta,
title={DEBERTA: DECODING - ENHANCED BERT WITH DISENTANGLED ATTENTION},
author={Pengcheng He and Xiaodong Liu and Jianfeng Gao and Weizhu Chen},
booktitle={International Conference on Learning Representations},
year={2021},
url={https://openreview.net/forum?id=XPZIaotutsD}
}