1epochv3

🚀 DeBERTa: Decoding-enhanced BERT with Disentangled Attention

DeBERTa is a model that improves upon BERT and RoBERTa by using disentangled attention and an enhanced mask decoder. It outperforms BERT and RoBERTa in most NLU tasks with 80GB of training data. This README provides details about the DeBERTa V2 xxlarge model and its performance on various NLU tasks.

✨ Features

• Enhanced Architecture: DeBERTa uses disentangled attention and an enhanced mask decoder to improve upon BERT and RoBERTa.
• High Performance: It outperforms BERT and RoBERTa in most NLU tasks with 80GB of training data.
• Multiple Sizes: Available in different sizes, including Large, XLarge, V2-XLarge, and V2-XXLarge.

📦 Installation

To run the DeBERTa V2-XXLarge model, you need to install the necessary dependencies. You can install them using the following commands:

pip install datasets
pip install deepspeed

💻 Usage Examples

Basic Usage

To run the model with Deepspeed, you can use the following commands:

# Download the deepspeed config file
wget https://huggingface.co/microsoft/deberta-v2-xxlarge/resolve/main/ds_config.json -O ds_config.json
export TASK_NAME=mnli
output_dir="ds_results"
num_gpus=8
batch_size=8
python -m torch.distributed.launch --nproc_per_node=${num_gpus} \
  run_glue.py \
  --model_name_or_path microsoft/deberta-v2-xxlarge \
  --task_name $TASK_NAME \
  --do_train \
  --do_eval \
  --max_seq_length 256 \
  --per_device_train_batch_size ${batch_size} \
  --learning_rate 3e-6 \
  --num_train_epochs 3 \
  --output_dir $output_dir \
  --overwrite_output_dir \
  --logging_steps 10 \
  --logging_dir $output_dir \
  --deepspeed ds_config.json

Advanced Usage

You can also run the model with --sharded_ddp using the following commands:

cd transformers/examples/text-classification/
export TASK_NAME=mnli
python -m torch.distributed.launch --nproc_per_node=8 run_glue.py \
--model_name_or_path microsoft/deberta-v2-xxlarge \
--task_name $TASK_NAME \
--do_train \
--do_eval \
--max_seq_length 256 \
--per_device_train_batch_size 8 \
--learning_rate 3e-6 \
--num_train_epochs 3 \
--output_dir /tmp/$TASK_NAME/ \
--overwrite_output_dir \
--sharded_ddp \
--fp16

📚 Documentation

Fine-tuning on NLU tasks

The following table shows the dev results on SQuAD 1.1/2.0 and several GLUE benchmark tasks:

Notes

• ¹ Following RoBERTa, for RTE, MRPC, STS-B, we fine-tune the tasks based on DeBERTa-Large-MNLI, DeBERTa-XLarge-MNLI, DeBERTa-V2-XLarge-MNLI, DeBERTa-V2-XXLarge-MNLI. The results of SST-2/QQP/QNLI/SQuADv2 will also be slightly improved when start from MNLI fine-tuned models, however, we only report the numbers fine-tuned from pretrained base models for those 4 tasks.
• ² To try the XXLarge model with HF transformers, we recommand using deepspeed as it's faster and saves memory.

Citation

If you find DeBERTa useful for your work, please cite the following paper:

@inproceedings{
he2021deberta,
title={DEBERTA: DECODING-ENHANCED BERT WITH DISENTANGLED ATTENTION},
author={Pengcheng He and Xiaodong Liu and Jianfeng Gao and Weizhu Chen},
booktitle={International Conference on Learning Representations},
year={2021},
url={https://openreview.net/forum?id=XPZIaotutsD}
}

📄 License

This project is licensed under the MIT License.