Deberta Base

🚀 DeBERTa: Decoding-enhanced BERT with Disentangled Attention

DeBERTa improves the BERT and RoBERTa models using disentangled attention and enhanced mask decoder. It offers superior performance on most NLU tasks with 80GB of training data.

🚀 Quick Start

DeBERTa enhances the BERT and RoBERTa models by leveraging disentangled attention and an enhanced mask decoder. With 80GB of training data, it outperforms BERT and RoBERTa on the majority of NLU tasks.

For more details and updates, please visit the official repository.

✨ Features

Fine-tuning on NLU tasks

We present the dev results on SQuAD 1.1/2.0 and MNLI tasks.

📄 License

This project is licensed under the MIT license.

📚 Documentation

Citation

If you find DeBERTa useful for your work, please cite the following paper:

@inproceedings{
he2021deberta,
title={DEBERTA: DECODING-ENHANCED BERT WITH DISENTANGLED ATTENTION},
author={Pengcheng He and Xiaodong Liu and Jianfeng Gao and Weizhu Chen},
booktitle={International Conference on Learning Representations},
year={2021},
url={https://openreview.net/forum?id=XPZIaotutsD}
}