Deberta V2 Xxlarge

🚀 DeBERTa: Decoding-enhanced BERT with Disentangled Attention

DeBERTa improves upon the BERT and RoBERTa models by leveraging disentangled attention and an enhanced mask decoder. With 80GB of training data, it outperforms BERT and RoBERTa in the majority of NLU tasks.

For more details and updates, please visit the official repository.

This is the DeBERTa V2 xxlarge model, featuring 48 layers and a hidden size of 1536. It has a total of 1.5B parameters and was trained on 160GB of raw data.

🚀 Quick Start

This section provides an overview of the model and how to use it for fine - tuning on NLU tasks.

✨ Features

• Disentangled Attention: DeBERTa uses disentangled attention to improve the performance of the model.
• Enhanced Mask Decoder: It has an enhanced mask decoder that further boosts the model's capabilities.
• High Performance: Outperforms BERT and RoBERTa on most NLU tasks with appropriate training data.

📦 Installation

To run the model with Deepspeed, you need to install the following dependencies:

pip install datasets
pip install deepspeed

# Download the deepspeed config file
wget https://huggingface.co/microsoft/deberta-v2-xxlarge/resolve/main/ds_config.json -O ds_config.json

💻 Usage Examples

Basic Usage

Here is an example of running the model with Deepspeed on the mnli task:

export TASK_NAME=mnli
output_dir="ds_results"
num_gpus=8
batch_size=8
python -m torch.distributed.launch --nproc_per_node=${num_gpus} \
  run_glue.py \
  --model_name_or_path microsoft/deberta-v2-xxlarge \
  --task_name $TASK_NAME \
  --do_train \
  --do_eval \
  --max_seq_length 256 \
  --per_device_train_batch_size ${batch_size} \
  --learning_rate 3e-6 \
  --num_train_epochs 3 \
  --output_dir $output_dir \
  --overwrite_output_dir \
  --logging_steps 10 \
  --logging_dir $output_dir \
  --deepspeed ds_config.json

Advanced Usage

You can also run the model with --sharded_ddp:

cd transformers/examples/text-classification/
export TASK_NAME=mnli
python -m torch.distributed.launch --nproc_per_node=8 run_glue.py   --model_name_or_path microsoft/deberta-v2-xxlarge   \
--task_name $TASK_NAME   --do_train   --do_eval   --max_seq_length 256   --per_device_train_batch_size 8   \
--learning_rate 3e-6   --num_train_epochs 3   --output_dir /tmp/$TASK_NAME/ --overwrite_output_dir --sharded_ddp --fp16

📚 Documentation

Fine - tuning on NLU tasks

The following table presents the dev results on SQuAD 1.1/2.0 and several GLUE benchmark tasks:

Notes

• ¹ Following RoBERTa, for RTE, MRPC, STS - B, we fine - tune the tasks based on DeBERTa - Large - MNLI, DeBERTa - XLarge - MNLI, DeBERTa - V2 - XLarge - MNLI, DeBERTa - V2 - XXLarge - MNLI. The results of SST - 2/QQP/QNLI/SQuADv2 will also be slightly improved when starting from MNLI fine - tuned models. However, we only report the numbers fine - tuned from pretrained base models for those 4 tasks.
• ² To try the XXLarge model with HF transformers, we recommend using deepspeed as it's faster and saves memory.

Citation

If you find DeBERTa useful for your work, please cite the following paper:

@inproceedings{
he2021deberta,
title={DEBERTA: DECODING - ENHANCED BERT WITH DISENTANGLED ATTENTION},
author={Pengcheng He and Xiaodong Liu and Jianfeng Gao and Weizhu Chen},
booktitle={International Conference on Learning Representations},
year={2021},
url={https://openreview.net/forum?id=XPZIaotutsD}
}

📄 License

This project is licensed under the MIT license.