Gpt2 Chinese Ancient

🚀 Chinese Ancient GPT2 Model

This model is designed to generate ancient Chinese text, pre - trained using advanced pre - training frameworks.

🚀 Quick Start

The Chinese Ancient GPT2 Model is pre - trained by [UER - py](https://github.com/dbiir/UER - py/), introduced in this paper. Additionally, it can also be pre - trained by TencentPretrain introduced in this paper, which inherits UER - py to support models with over one billion parameters and extends it to a multimodal pre - training framework.

You can download the model from the [UER - py Modelzoo page](https://github.com/dbiir/UER - py/wiki/Modelzoo), [GPT2 - Chinese Github page](https://github.com/Morizeyao/GPT2 - Chinese), or via HuggingFace from the link [gpt2 - chinese - ancient](https://huggingface.co/uer/gpt2 - chinese - ancient).

✨ Features

• The model is specifically used for generating ancient Chinese text.
• It can be pre - trained by multiple pre - training frameworks, providing flexibility.

💻 Usage Examples

Basic Usage

>>> from transformers import BertTokenizer, GPT2LMHeadModel, TextGenerationPipeline
>>> tokenizer = BertTokenizer.from_pretrained("uer/gpt2-chinese-ancient")
>>> model = GPT2LMHeadModel.from_pretrained("uer/gpt2-chinese-ancient")
>>> text_generator = TextGenerationPipeline(model, tokenizer)   
>>> text_generator("当是时", max_length=100, do_sample=True)
    [{'generated_text': '[CLS]当是时 所 议 者 不 为 无 据 ， 况 亦 在 之 列 乎 ？ 然 则 今 日 之 事 ， 所 当 思 者 在 何 ？ 欲 求 国 是 于 天 下 ， 莫 在 于 得 人 。 臣 以 为 求 人 之 法 ， 不 在 多 用 官 一 途 。 诚 使 得 才 者 众 ， 人 才 者 优 ， 则 治 所 当 得 ， 而 不 事 于 官 者 ， 人 才 乃 其 常 也 。 所 当 讲 者'}]

📚 Documentation

Training Data

The training data contains 3,000,000 ancient Chinese texts collected by daizhigev20. Since part of the ancient corpus has no punctuation, the ancient Chinese punctuation system developed by BNU ICIP lab was used.

Training Procedure

The model is pre - trained by [UER - py](https://github.com/dbiir/UER - py/) on Tencent Cloud. Here are the steps:

1. Pre - process the data:

python3 preprocess.py --corpus_path corpora/ancient_chinese.txt \
                      --vocab_path models/google_zh_ancient_vocab.txt \
                      --dataset_path ancient_chinese_dataset.pt --processes_num 16 \
                      --seq_length 320 --data_processor lm

2. Pre - train the model:

python3 pretrain.py --dataset_path ancient_chinese_dataset.pt \
                    --vocab_path models/google_zh_ancient_vocab.txt \
                    --config_path models/bert_base_config.json \
                    --output_model_path models/ancient_chinese_gpt2_model.bin \
                    --world_size 8 --gpu_ranks 0 1 2 3 4 5 6 7 \
                    --total_steps 500000 --save_checkpoint_steps 100000 --report_steps 10000 \
                    --learning_rate 5e-4 --batch_size 32

3. Convert the pre - trained model into Huggingface's format:

python3 scripts/convert_gpt2_from_uer_to_huggingface.py --input_model_path models/ancient_chinese_gpt2_model.bin-500000 \
                                                        --output_model_path pytorch_model.bin \
                                                        --layers_num 12

BibTeX entry and citation info

@article{radford2019language,
  title={Language Models are Unsupervised Multitask Learners},
  author={Radford, Alec and Wu, Jeff and Child, Rewon and Luan, David and Amodei, Dario and Sutskever, Ilya},
  year={2019}
}

@article{zhao2019uer,
  title={UER: An Open-Source Toolkit for Pre-training Models},
  author={Zhao, Zhe and Chen, Hui and Zhang, Jinbin and Zhao, Xin and Liu, Tao and Lu, Wei and Chen, Xi and Deng, Haotang and Ju, Qi and Du, Xiaoyong},
  journal={EMNLP-IJCNLP 2019},
  pages={241},
  year={2019}
}

@article{zhao2023tencentpretrain,
  title={TencentPretrain: A Scalable and Flexible Toolkit for Pre-training Models of Different Modalities},
  author={Zhao, Zhe and Li, Yudong and Hou, Cheng and Zhao, Jing and others},
  journal={ACL 2023},
  pages={217},
  year={2023}
}