Ernie 2.0 Base En

🚀 ERNIE-2.0

ERNIE 2.0 is a continual pre-training framework that enhances language understanding through incremental pre-training tasks.

🚀 Quick Start

ERNIE 2.0 is a continual pre-training framework proposed by Baidu in 2019. It builds and learns incrementally pre-training tasks through constant multi-task learning. Experimental results show that ERNIE 2.0 outperforms BERT and XLNet on 16 tasks, including English tasks on GLUE benchmarks and several common Chinese tasks.

For more details, refer to: https://arxiv.org/abs/1907.12412

✨ Features

This released PyTorch model is converted from the officially released PaddlePaddle ERNIE model. A series of experiments have been carried out to verify the accuracy of the conversion.

• Official PaddlePaddle ERNIE repo: https://github.com/PaddlePaddle/ERNIE
• PyTorch Conversion repo: https://github.com/nghuyong/ERNIE-Pytorch

💻 Usage Examples

Basic Usage

from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("nghuyong/ernie-2.0-base-en")
model = AutoModel.from_pretrained("nghuyong/ernie-2.0-base-en")

📚 Documentation

Citation

@article{sun2019ernie20,
  title={ERNIE 2.0: A Continual Pre-training Framework for Language Understanding},
  author={Sun, Yu and Wang, Shuohuan and Li, Yukun and Feng, Shikun and Tian, Hao and Wu, Hua and Wang, Haifeng},
  journal={arXiv preprint arXiv:1907.12412},
  year={2019} 
}