Chinese Text Correction 1.5b

🚀 Chinese Text Correction Model

A Chinese text correction model for spelling and grammar error correction.

🚀 Quick Start

The Chinese text correction model chinese-text-correction-1.5b is designed for spelling and grammar error correction.

Evaluation of test data for shibing624/chinese-text-correction-1.5b:

The overall performance of CSC test:

✨ Features

Model Information

Available Models

Evaluation Results

• Evaluation Metrics: F1
• CSC (Chinese Spelling Correction): A spelling correction model that can handle errors such as similar - sounding, similar - shaped characters, and grammar errors with aligned lengths.
• CTC (Chinese Text Correction): A text correction model that supports the correction of spelling and grammar errors with aligned lengths, and can also handle errors with unaligned lengths such as extra or missing characters.
• GPU: Tesla V100, 32 GB of video memory

📦 Installation

Install pycorrector

pip install -U pycorrector

Install HuggingFace Transformers

pip install transformers 

💻 Usage Examples

Usage with pycorrector

This project is open - sourced in the pycorrector project: pycorrector. It supports using fine - tuned large models for text correction. You can call it with the following commands:

from pycorrector.gpt.gpt_corrector import GptCorrector

if __name__ == '__main__':
    error_sentences = [
        '真麻烦你了。希望你们好好的跳无',
        '少先队员因该为老人让坐',
        '机七学习是人工智能领遇最能体现智能的一个分知',
        '一只小鱼船浮在平净的河面上',
        '我的家乡是有明的渔米之乡',
    ]
    m = GptCorrector("shibing624/chinese-text-correction-1.5b")

    batch_res = m.correct_batch(error_sentences)
    for i in batch_res:
        print(i)
        print()

Usage with HuggingFace Transformers

Without pycorrector, you can use the model like this:

# pip install transformers
from transformers import AutoModelForCausalLM, AutoTokenizer
checkpoint = "shibing624/chinese-text-correction-1.5b"

device = "cuda" # for GPU usage or "cpu" for CPU usage
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForCausalLM.from_pretrained(checkpoint).to(device)

input_content = "文本纠错：\n少先队员因该为老人让坐。"

messages = [{"role": "user", "content": input_content}]
input_text=tokenizer.apply_chat_template(messages, tokenize=False)

print(input_text)

inputs = tokenizer.encode(input_text, return_tensors="pt").to(device)
outputs = model.generate(inputs, max_new_tokens=1024, temperature=0, do_sample=False, repetition_penalty=1.08)

print(tokenizer.decode(outputs[0]))

Output:

少先队员应该为老人让座。

📚 Documentation

Model File Structure

shibing624/chinese-text-correction-1.5b
|-- added_tokens.json
|-- config.json
|-- generation_config.json
|-- merges.txt
|-- model.safetensors
|-- model.safetensors.index.json
|-- README.md
|-- special_tokens_map.json
|-- tokenizer_config.json
|-- tokenizer.json
`-- vocab.json

Training Parameters

• num_epochs: 8
• batch_size: 4
• steps: 36000
• eval_loss: 0.14
• base model: Qwen/Qwen2.5 - 1.5B - Instruct
• train data: shibing624/chinese_text_correction
• train time: 9 days 8 hours
• eval_loss:
• train_loss:

Training Datasets

Chinese Error Correction Dataset

• Data: shibing624/chinese_text_correction

If you need to train a Qwen error correction model, please refer to https://github.com/shibing624/pycorrector or https://github.com/shibing624/MedicalGPT.

📄 License

This project is licensed under the apache - 2.0 license.

📚 Citation

@software{pycorrector,
  author = {Xu Ming},
  title = {pycorrector: Implementation of language model finetune},
  year = {2024},
  url = {https://github.com/shibing624/pycorrector},
}