Chattcm 7B SFT

🚀 ChatTCM-7B-SFT

A fine - tuned model based on SylvanL/ChatTCM-7B-Pretrain, designed for Chinese medicine text generation with enhanced capabilities.

🚀 Quick Start

The test evaluation results are on the way... The model of the second epoch is also on the way...

On 2 A800 - 80G GPUs, based on the SylvanL/ChatTCM-7B-Pretrain model and using the llamafactory framework, a full - parameter supervised fine - tuning (full Supervised Fine - tuning) for 1 epoch was conducted with the SylvanL/Traditional-Chinese-Medicine-Dataset-SFT.

Under the premise of no obvious instruction loss or catastrophic forgetting, the model has been endowed with the following capabilities:

1. It can translate classical Chinese/archaic Chinese into modern Chinese to enhance the understanding and utilization of traditional Chinese medicine classics.
2. It has the clinical diagnosis logic and prescription - recommending ability similar to mainstream - school practicing doctors, capable of understanding the input patient conditions and making judgments and analyses.
3. It has good Chinese medicine knowledge - answering ability, providing comprehensive and reliable answers to knowledge points in the field of Chinese medicine.
4. It strengthens the model's basic NLP ability for Chinese medicine terms, better enabling general functions such as Chinese medicine named entity recognition, relationship extraction, relevance analysis, synonymous entity disambiguation, spelling check and correction.

P.S.: No identification has been implanted into the model.

💻 Usage Examples

Basic Usage

# Optional Instruction
将输入的古文翻译成现代文。
请为输入的现代文找到其对应的古文原文与出处。
基于输入的患者医案记录，直接给出你的证型诊断，无需给出原因。
基于输入的患者医案记录，直接给出你的疾病诊断，无需给出原因。
基于输入的患者医案记录，直接给出你认为的方剂中药组成。
基于输入的患者医案记录，直接给出你认为的【治疗方案】{可多选}∈["中药", "成药", "方剂"]，和【诊断】{可多选}∈["证型", "治法", "西医诊断", "中医诊断"]：

Advanced Usage

# Epoch 1 training information
epoch 1: 
    "num_input_tokens_seen": 1649269888,
    "total_flos": 3298213988794368.0,
    "train_loss": 1.0691444667014194,
    "train_runtime": 587389.2072,
    "train_samples_per_second": 3.483,
    "train_steps_per_second": 0.016

# Training command
llamafactory-cli train \
    --stage sft \
    --do_train True \
    --model_name_or_path {SylvanL/ChatTCM-7B-Pretrain} \
    --preprocessing_num_workers 16 \
    --finetuning_type full \
    --template default \
    --flash_attn auto \
    --dataset_dir {dataset_dir} \
    --dataset SFT_medicalKnowledge_source1_548404,SFT_medicalKnowledge_source2_99334,SFT_medicalKnowledge_source3_556540,SFT_nlpDiseaseDiagnosed_61486,SFT_nlpSyndromeDiagnosed_48665,SFT_structGeneral_310860,SFT_structPrescription_92896,_SFT_traditionalTrans_1959542.json,{BAAI/COIG},{m-a-p/COIG-CQIA} \
    --cutoff_len 1024 \
    --learning_rate 5e-05 \
    --num_train_epochs 1.0 \
    --max_samples 1000000 \
    --per_device_train_batch_size 28 \
    --gradient_accumulation_steps 4 \
    --lr_scheduler_type cosine \
    --max_grad_norm 1.0 \
    --logging_steps 1 \
    --save_steps 1000 \
    --warmup_steps 0 \
    --optim adamw_torch \
    --packing False \
    --report_to none \
    --output_dir {output_dir} \
    --bf16 True \
    --plot_loss True \
    --ddp_timeout 180000000 \
    --include_num_input_tokens_seen True \
    --deepspeed cache/ds_z3_offload_config.json

📄 License

This project is licensed under the apache - 2.0 license.

📋 Information Table