Deberta Base

🚀 DeBERTa: Decoding-enhanced BERT with Disentangled Attention

DeBERTa improves the BERT and RoBERTa models using disentangled attention and enhanced mask decoder, offering better performance on most NLU tasks with 80GB training data.

🚀 Quick Start

DeBERTa enhances the BERT and RoBERTa models by leveraging disentangled attention and an enhanced mask decoder. It outperforms BERT and RoBERTa in the majority of NLU tasks when trained with 80GB of data.

For more detailed information and the latest updates, please visit the official repository.

✨ Features

Fine-tuning on NLU tasks

We present the development results on SQuAD 1.1/2.0 and MNLI tasks.

📄 License

This project is licensed under the MIT license.

📚 Documentation

Citation

If you find DeBERTa useful for your work, please cite the following paper:

@inproceedings{
he2021deberta,
title={DEBERTA: DECODING-ENHANCED BERT WITH DISENTANGLED ATTENTION},
author={Pengcheng He and Xiaodong Liu and Jianfeng Gao and Weizhu Chen},
booktitle={International Conference on Learning Representations},
year={2021},
url={https://openreview.net/forum?id=XPZIaotutsD}
}