Mengzi T5 Base Chinese Correction

🚀 T5 for Chinese Spelling Correction Model

This is a Chinese spelling correction model that achieves near-SOTA performance on the SIGHAN2015 test set.

The model shibing624/mengzi-t5-base-chinese-correction evaluated on the SIGHAN2015 test data shows the following results:

• Sentence Level: precision: 0.8321, recall: 0.6390, f1: 0.7229

The model was trained on the "SIGHAN + Wang271K Chinese correction dataset" provided below, and it reaches a near-SOTA level on the SIGHAN2015 test set.

The model structure remains unchanged. After fine-tuning on the Chinese correction dataset, the evaluation of the correction effect is excellent, indicating great potential of the model.

🚀 Quick Start

✨ Features

• This project is open - sourced in the Chinese text correction project pycorrector, which supports the T5 model.

📦 Installation

You can install the necessary package by running the following command:

pip install -U pycorrector

💻 Usage Examples

Basic Usage

from pycorrector.t5.t5_corrector import T5Corrector
nlp = T5Corrector("shibing624/mengzi-t5-base-chinese-correction").batch_t5_correct
i = "今天新情很好"
print(i, ' => ', nlp([i]))

The output will be:

今天新情很好  =>  今天心情很好 [('新', '心', 2, 3)]

Model File Composition

mengzi-t5-base-chinese-correction
|-- config.json
|-- pytorch_model.bin
|-- special_tokens_map.json
|-- spiece.model
|-- tokenizer_config.json
`-- tokenizer.json

If you need to train the t5 - correction model, please refer to https://github.com/shibing624/pycorrector/tree/master/pycorrector/t5

Training Datasets

SIGHAN + Wang271K Chinese Correction Dataset

The data format of the SIGHAN + Wang271K Chinese correction dataset is as follows:

[
    {
        "id": "B2-4029-3",
        "original_text": "晚间会听到嗓音，白天的时候大家都不会太在意，但是在睡觉的时候这嗓音成为大家的恶梦。",
        "wrong_ids": [
            5,
            31
        ],
        "correct_text": "晚间会听到噪音，白天的时候大家都不会太在意，但是在睡觉的时候这噪音成为大家的恶梦。"
    }
]

📄 License

This project is under the Apache - 2.0 license.

📚 Documentation

Citation

@software{pycorrector,
  author = {Xu Ming},
  title = {pycorrector: Text Error Correction Tool},
  year = {2021},
  url = {https://github.com/shibing624/pycorrector},
}