Rbt6

🚀 Re-trained 6-layer RoBERTa-wwm-ext Model

This is a re-trained 6-layer RoBERTa-wwm-ext model designed to further accelerate Chinese natural language processing.

🚀 Quick Start

This repository provides a pre-trained Chinese BERT model with Whole Word Masking, aiming to boost the efficiency of Chinese natural language processing.

The research paper Pre-Training with Whole Word Masking for Chinese BERT is written by Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Ziqing Yang, Shijin Wang, and Guoping Hu.

This repository is developed based on: https://github.com/google-research/bert

You may also be interested in the following related projects:

• Chinese BERT series: https://github.com/ymcui/Chinese-BERT-wwm
• Chinese MacBERT: https://github.com/ymcui/MacBERT
• Chinese ELECTRA: https://github.com/ymcui/Chinese-ELECTRA
• Chinese XLNet: https://github.com/ymcui/Chinese-XLNet
• Knowledge Distillation Toolkit - TextBrewer: https://github.com/airaria/TextBrewer

More resources by HFL: https://github.com/ymcui/HFL-Anthology

📚 Documentation

Citation

If you find the technical report or resource useful, please cite the following technical reports in your paper:

• Primary: https://arxiv.org/abs/2004.13922

@inproceedings{cui-etal-2020-revisiting,
    title = "Revisiting Pre-Trained Models for {C}hinese Natural Language Processing",
    author = "Cui, Yiming  and
      Che, Wanxiang  and
      Liu, Ting  and
      Qin, Bing  and
      Wang, Shijin  and
      Hu, Guoping",
    booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings",
    month = nov,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.findings-emnlp.58",
    pages = "657--668",
}

• Secondary: https://arxiv.org/abs/1906.08101

@article{chinese-bert-wwm,
  title={Pre-Training with Whole Word Masking for Chinese BERT},
  author={Cui, Yiming and Che, Wanxiang and Liu, Ting and Qin, Bing and Yang, Ziqing and Wang, Shijin and Hu, Guoping},
  journal={arXiv preprint arXiv:1906.08101},
  year={2019}
 }

📄 License

This project is licensed under the Apache-2.0 license.