Rbtl3

🚀 This is a re-trained 3-layer RoBERTa-wwm-ext-large model.

This project provides a re-trained 3-layer RoBERTa-wwm-ext-large model to accelerate Chinese natural language processing.

🚀 Quick Start

This model aims to further accelerate Chinese natural language processing by offering Chinese pre-trained BERT with Whole Word Masking.

Pre-Training with Whole Word Masking for Chinese BERT
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Ziqing Yang, Shijin Wang, Guoping Hu

This repository is developed based on: Google's BERT repository

You may also be interested in the following related projects:

• Chinese BERT series
• Chinese MacBERT
• Chinese ELECTRA
• Chinese XLNet
• Knowledge Distillation Toolkit - TextBrewer

More resources by HFL can be found at: HFL-Anthology

📚 Documentation

Citation

If you find the technical report or resource useful, please cite the following technical reports in your paper:

• Primary: Revisiting Pre-Trained Models for Chinese Natural Language Processing

@inproceedings{cui-etal-2020-revisiting,
    title = "Revisiting Pre-Trained Models for {C}hinese Natural Language Processing",
    author = "Cui, Yiming  and
      Che, Wanxiang  and
      Liu, Ting  and
      Qin, Bing  and
      Wang, Shijin  and
      Hu, Guoping",
    booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings",
    month = nov,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.findings-emnlp.58",
    pages = "657--668",
}

• Secondary: Pre-Training with Whole Word Masking for Chinese BERT

@article{chinese-bert-wwm,
  title={Pre-Training with Whole Word Masking for Chinese BERT},
  author={Cui, Yiming and Che, Wanxiang and Liu, Ting and Qin, Bing and Yang, Ziqing and Wang, Shijin and Hu, Guoping},
  journal={arXiv preprint arXiv:1906.08101},
  year={2019}
 }

📄 License

This project is licensed under the Apache-2.0 license.