Debertav3 Mnli Snli Anli

🚀 DeBERTa: Decoding-enhanced BERT with Disentangled Attention

DeBERTa improves upon the BERT and RoBERTa models by leveraging disentangled attention and an enhanced mask decoder. Trained with 80GB of data, it outperforms its predecessors in most NLU tasks.

For more details and updates, please visit the official repository. This is the DeBERTa V2 xxlarge model, featuring 48 layers and a 1536 hidden size. With a total of 1.5B parameters, it was trained on 160GB of raw data.

🚀 Quick Start

This section provides an overview of the DeBERTa model and its performance on various NLU tasks. You can find detailed installation and usage instructions below.

✨ Features

• Disentangled Attention: DeBERTa uses disentangled attention to improve the understanding of language structure.
• Enhanced Mask Decoder: The enhanced mask decoder helps in better prediction of masked tokens.
• High Performance: Outperforms BERT and RoBERTa on most NLU tasks.

📦 Installation

To run the DeBERTa model, you need to install the necessary dependencies. You can use Deepspeed or --sharded_ddp for training.

Install with Deepspeed

pip install datasets
pip install deepspeed
# Download the deepspeed config file
wget https://huggingface.co/microsoft/deberta-v2-xxlarge/resolve/main/ds_config.json -O ds_config.json

Install with --sharded_ddp

cd transformers/examples/text-classification/

💻 Usage Examples

Run with Deepspeed

export TASK_NAME=mnli
output_dir="ds_results"
num_gpus=8
batch_size=8
python -m torch.distributed.launch --nproc_per_node=${num_gpus} \
  run_glue.py \
  --model_name_or_path microsoft/deberta-v2-xxlarge \
  --task_name $TASK_NAME \
  --do_train \
  --do_eval \
  --max_seq_length 256 \
  --per_device_train_batch_size ${batch_size} \
  --learning_rate 3e-6 \
  --num_train_epochs 3 \
  --output_dir $output_dir \
  --overwrite_output_dir \
  --logging_steps 10 \
  --logging_dir $output_dir \
  --deepspeed ds_config.json

Run with --sharded_ddp

export TASK_NAME=mnli
python -m torch.distributed.launch --nproc_per_node=8 run_glue.py   --model_name_or_path microsoft/deberta-v2-xxlarge   \
--task_name $TASK_NAME   --do_train   --do_eval   --max_seq_length 256   --per_device_train_batch_size 8   \
--learning_rate 3e-6   --num_train_epochs 3   --output_dir /tmp/$TASK_NAME/ --overwrite_output_dir --sharded_ddp --fp16

📚 Documentation

Fine-tuning on NLU tasks

The following table shows the dev results on SQuAD 1.1/2.0 and several GLUE benchmark tasks.

Notes

• ¹ Following RoBERTa, for RTE, MRPC, STS-B, we fine-tune the tasks based on DeBERTa-Large-MNLI, DeBERTa-XLarge-MNLI, DeBERTa-V2-XLarge-MNLI, DeBERTa-V2-XXLarge-MNLI. The results of SST-2/QQP/QNLI/SQuADv2 will also be slightly improved when starting from MNLI fine-tuned models. However, we only report the numbers fine-tuned from pretrained base models for those 4 tasks.
• ² To try the XXLarge model with HF transformers, we recommend using deepspeed as it's faster and saves memory.

📄 License

No license information provided in the original document.

🔧 Technical Details

• Model Type: Decoding-enhanced BERT with Disentangled Attention
• Training Data: 80GB for the base model, 160GB for the DeBERTa V2 xxlarge model

📖 Citation

If you find DeBERTa useful for your work, please cite the following paper:

@inproceedings{
he2021deberta,
title={DEBERTA: DECODING-ENHANCED BERT WITH DISENTANGLED ATTENTION},
author={Pengcheng He and Xiaodong Liu and Jianfeng Gao and Weizhu Chen},
booktitle={International Conference on Learning Representations},
year={2021},
url={https://openreview.net/forum?id=XPZIaotutsD}
}

💡 Usage Tip

To try the XXLarge model with HF transformers, we recommend using deepspeed as it's faster and saves memory.