Chinese Roberta L 8 H 256

🚀 Chinese RoBERTa Miniatures

This project offers a set of 24 Chinese RoBERTa models. These models are pre - trained to effectively handle various Chinese language processing tasks, providing high - quality solutions for natural language processing in the Chinese context.

🚀 Quick Start

You can use this model directly with a pipeline for masked language modeling (take the case of RoBERTa - Medium):

>>> from transformers import pipeline
>>> unmasker = pipeline('fill - mask', model='uer/chinese_roberta_L - 8_H - 512')
>>> unmasker("中国的首都是[MASK]京。")
[
    {'sequence': '[CLS] 中 国 的 首 都 是 北 京 。 [SEP]', 
     'score': 0.8701988458633423, 
     'token': 1266, 
     'token_str': '北'},
    {'sequence': '[CLS] 中 国 的 首 都 是 南 京 。 [SEP]',
     'score': 0.1194809079170227, 
     'token': 1298, 
     'token_str': '南'},
    {'sequence': '[CLS] 中 国 的 首 都 是 东 京 。 [SEP]', 
     'score': 0.0037803512532263994, 
     'token': 691, 
     'token_str': '东'},
    {'sequence': '[CLS] 中 国 的 首 都 是 普 京 。 [SEP]',
     'score': 0.0017127094324678183, 
     'token': 3249,
     'token_str': '普'},
    {'sequence': '[CLS] 中 国 的 首 都 是 望 京 。 [SEP]',
     'score': 0.001687526935711503,
     'token': 3307, 
     'token_str': '望'}
]

Here is how to use this model to get the features of a given text in PyTorch:

from transformers import BertTokenizer, BertModel
tokenizer = BertTokenizer.from_pretrained('uer/chinese_roberta_L - 8_H - 512')
model = BertModel.from_pretrained("uer/chinese_roberta_L - 8_H - 512")
text = "用你喜欢的任何文本替换我。"
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)

and in TensorFlow:

from transformers import BertTokenizer, TFBertModel
tokenizer = BertTokenizer.from_pretrained('uer/chinese_roberta_L - 8_H - 512')
model = TFBertModel.from_pretrained("uer/chinese_roberta_L - 8_H - 512")
text = "用你喜欢的任何文本替换我。"
encoded_input = tokenizer(text, return_tensors='tf')
output = model(encoded_input)

✨ Features

• Multiple Model Sizes: There are 24 different Chinese RoBERTa models with various combinations of layer numbers (L) and hidden layer dimensions (H), including Tiny, Mini, Small, Medium, and Base models, which can meet different performance and resource requirements.
• High Performance: These models achieve good scores on the development sets of six Chinese tasks, such as book review, sentiment analysis, sentence matching, news classification, and natural language inference.
• Open - source and Reproducible: The models are pre - trained using publicly available corpora, and all training details are provided, facilitating users to reproduce the results.

📦 Installation

The installation process is mainly about using the pre - trained models in the transformers library. You can install the transformers library via the following command:

pip install transformers

💻 Usage Examples

Basic Usage

from transformers import pipeline
unmasker = pipeline('fill - mask', model='uer/chinese_roberta_L - 8_H - 512')
result = unmasker("中国的首都是[MASK]京。")
print(result)

Advanced Usage

# Use in a more complex NLP task, such as fine - tuning on a custom dataset
from transformers import BertTokenizer, BertForMaskedLM
import torch

tokenizer = BertTokenizer.from_pretrained('uer/chinese_roberta_L - 8_H - 512')
model = BertForMaskedLM.from_pretrained("uer/chinese_roberta_L - 8_H - 512")

# Prepare data
texts = ["这是一个测试句子。", "另一个测试句子。"]
inputs = tokenizer(texts, return_tensors='pt', padding=True, truncation=True)

# Forward pass
outputs = model(**inputs)
predictions = outputs.logits

📚 Documentation

Model Description

This is the set of 24 Chinese RoBERTa models pre - trained by [UER - py](https://github.com/dbiir/UER - py/), which is introduced in this paper. Besides, the models could also be pre - trained by TencentPretrain introduced in this paper, which inherits UER - py to support models with parameters above one billion, and extends it to a multimodal pre - training framework.

Turc et al. have shown that the standard BERT recipe is effective on a wide range of model sizes. Following their paper, we released the 24 Chinese RoBERTa models. In order to facilitate users in reproducing the results, we used a publicly available corpus and provided all training details.

You can download the 24 Chinese RoBERTa miniatures either from the [UER - py Modelzoo page](https://github.com/dbiir/UER - py/wiki/Modelzoo), or via HuggingFace from the links below:

Here are scores on the development set of six Chinese tasks:

For each task, we selected the best fine - tuning hyperparameters from the lists below, and trained with the sequence length of 128:

• epochs: 3, 5, 8
• batch sizes: 32, 64
• learning rates: 3e - 5, 1e - 4, 3e - 4

🔧 Technical Details

Training Data

CLUECorpusSmall is used as training data. We found that models pre - trained on CLUECorpusSmall outperform those pre - trained on CLUECorpus2020, although CLUECorpus2020 is much larger than CLUECorpusSmall.

Training Procedure

Models are pre - trained by [UER - py](https://github.com/dbiir/UER - py/) on Tencent Cloud. We pre - train 1,000,000 steps with a sequence length of 128 and then pre - train 250,000 additional steps with a sequence length of 512. We use the same hyper - parameters on different model sizes.

Taking the case of RoBERTa - Medium

Stage 1:

python3 preprocess.py --corpus_path corpora/cluecorpussmall.txt \
                      --vocab_path models/google_zh_vocab.txt \
                      --dataset_path cluecorpussmall_seq128_dataset.pt \
                      --processes_num 32 --seq_length 128 \
                      --dynamic_masking --data_processor mlm

python3 pretrain.py --dataset_path cluecorpussmall_seq128_dataset.pt \
                    --vocab_path models/google_zh_vocab.txt \
                    --config_path models/bert/medium_config.json \
                    --output_model_path models/cluecorpussmall_roberta_medium_seq128_model.bin \
                    --world_size 8 --gpu_ranks 0 1 2 3 4 5 6 7 \
                    --total_steps 1000000 --save_checkpoint_steps 100000 --report_steps 50000 \
                    --learning_rate 1e - 4 --batch_size 64 \
                    --data_processor mlm --target mlm

Stage 2:

python3 preprocess.py --corpus_path corpora/cluecorpussmall.txt \
                      --vocab_path models/google_zh_vocab.txt \
                      --dataset_path cluecorpussmall_seq512_dataset.pt \
                      --processes_num 32 --seq_length 512 \
                      --dynamic_masking --data_processor mlm

python3 pretrain.py --dataset_path cluecorpussmall_seq512_dataset.pt \
                    --vocab_path models/google_zh_vocab.txt \
                    --pretrained_model_path models/cluecorpussmall_roberta_medium_seq128_model.bin - 1000000 \
                    --config_path models/bert/medium_config.json \
                    --output_model_path models/cluecorpussmall_roberta_medium_seq512_model.bin \
                    --world_size 8 --gpu_ranks 0 1 2 3 4 5 6 7 \
                    --total_steps 250000 --save_checkpoint_steps 50000 --report_steps 10000 \
                    --learning_rate 5e - 5 --batch_size 16 \
                    --data_processor mlm --target mlm

Finally, we convert the pre - trained model into Huggingface's format:

python3 scripts/convert_bert_from_uer_to_huggingface.py --input_model_path models/cluecorpussmall_roberta_medium_seq512_model.bin - 250000 \                                                        
                                                        --output_model_path pytorch_model.bin \
                                                        --layers_num 8 --type mlm

📄 License

BibTeX entry and citation info

@article{devlin2018bert,
  title={Bert: Pre-training of deep bidirectional transformers for language understanding},
  author={Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
  journal={arXiv preprint arXiv:1810.04805},
  year={2018}
}

@article{liu2019roberta,
  title={Roberta: A robustly optimized bert pretraining approach},
  author={Liu, Yinhan and Ott, Myle and Goyal, Naman and Du, Jingfei and Joshi, Mandar and Chen, Danqi and Levy, Omer and Lewis, Mike and Zettlemoyer, Luke and Stoyanov, Veselin},
  journal={arXiv preprint arXiv:1907.11692},
  year={2019}
}

@article{turc2019,
  title={Well-Read Students Learn Better: On the Importance of Pre-training Compact Models},
  author={Turc, Iulia and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
  journal={arXiv preprint arXiv:1908.08962v2 },
  year={2019}
}

@article{zhao2019uer,
  title={UER: An Open-Source Toolkit for Pre-training Models},
  author={Zhao, Zhe and Chen, Hui and Zhang, Jinbin and Zhao, Xin and Liu, Tao and Lu, Wei and Chen, Xi and Deng, Haotang and Ju, Qi and Du, Xiaoyong},
  journal={EMNLP-IJCNLP 2019},
  pages={241},
  year={2019}
}

@article{zhao2023tencentpretrain,
  title={TencentPretrain: A Scalable and Flexible Toolkit for Pre-training Models of Different Modalities},
  author={Zhao, Zhe and Li, Yudong and Hou, Cheng and Zhao, Jing and others},
  journal={ACL 2023},
  pages={217},
  year={2023}
}

[2_128]:https://huggingface.co/uer/chinese_roberta_L - 2_H - 128 [2_256]:https://huggingface.co/uer/chinese_roberta_L - 2_H - 256 [2_512]:https://huggingface.co/uer/chinese_roberta_L - 2_H - 512 [2_768]:https://huggingface.co/uer/chinese_roberta_L - 2_H - 768 [4_128]:https://huggingface.co/uer/chinese_roberta_L - 4_H - 128 [4_256]:https://huggingface.co/uer/chinese_roberta_L - 4_H - 256 [4_512]:https://huggingface.co/uer/chinese_roberta_L - 4_H - 512 [4_768]:https://huggingface.co/uer/chinese_roberta_L - 4_H - 768 [6_128]:https://huggingface.co/uer/chinese_roberta_L - 6_H - 128 [6_256]:https://huggingface.co/uer/chinese_roberta_L - 6_H - 256 [6_512]:https://huggingface.co/uer/chinese_roberta_L - 6_H - 512 [6_768]:https://huggingface.co/uer/chinese_roberta_L - 6_H - 768 [8_128]:https://huggingface.co/uer/chinese_roberta_L - 8_H - 128 [8_256]:https://huggingface.co/uer/chinese_roberta_L - 8_H - 256 [8_512]:https://huggingface.co/uer/chinese_roberta_L - 8_H - 512 [8_768]:https://huggingface.co/uer/chinese_roberta_L - 8_H - 768 [10_128]:https://huggingface.co/uer/chinese_roberta_L - 10_H - 128 [10_256]:https://huggingface.co/uer/chinese_roberta_L - 10_H - 256 [10_512]:https://huggingface.co/uer/chinese_roberta_L - 10_H - 512 [10_768]:https://huggingface.co/uer/chinese_roberta_L - 10_H - 768 [12_128]:https://huggingface.co/uer/chinese_roberta_L - 12_H - 128 [12_256]:https://huggingface.co/uer/chinese_roberta_L - 12_H - 256 [12_512]:https://huggingface.co/uer/chinese_roberta_L - 12_H - 512 [12_768]:https://huggingface.co/uer/chinese_roberta_L - 12_H - 768