Wingpt2 7B Base

🚀 WiNGPT2

WiNGPT2 is a GPT-based large model specifically designed for the medical vertical field. It aims to integrate professional medical knowledge, information, and data, providing intelligent information services such as medical Q&A, diagnostic support, and medical knowledge to the healthcare industry, thereby improving the efficiency of diagnosis and treatment and the quality of medical services.

🚀 Quick Start

WiNGPT (Winning Health's large medical language model, hereinafter referred to as WiNGPT) began its R & D and training in January 2023.

In March, Winning Health's AI laboratory completed the feasibility verification of WiNGPT - 001 and started internal testing. WiNGPT - 001 uses a general GPT architecture with 6 billion parameters and realizes the whole - process self - research from pre - training to fine - tuning.

In May this year, the training data volume of WiNGPT - 001 reached 9,720 items of drug knowledge, 18 drug types, more than 7,200 items of disease knowledge, more than 2,800 items of examination and inspection knowledge, knowledge from 53 books, and more than 1,100 guide documents. The total number of training tokens reached 3.7 billion.

In July, WiNGPT was upgraded to 7B and adopted the latest model architecture. It added the retrieval - enhanced generation ability and started the training of the 13B model and industry invitation testing.

In September, WiNGPT迎来最新版本迭代,推出了全新的WiNGPT2,新版本可以被轻松扩展和个性化并用于下游各种应用场景。

To give back to the open - source community, we open - sourced the WiNGPT2 - 7B version. Our original intention is to accelerate the common development of medical large language model technology and the industry through more open - source projects and ultimately benefit human health.

✨ Features

Core Functions

• Medical Knowledge Q&A: It can answer questions about medicine, health, diseases, etc., including but not limited to symptoms, treatments, drugs, prevention, and examinations.
• Natural Language Understanding: It can understand medical text information such as medical terms and medical records, and provide key information extraction and classification.
• Multi - turn Dialogue: It can play various medical professional roles such as doctors to have conversations with users and provide more accurate answers based on the context.
• Multi - task Support: It supports 32 medical tasks, 18 sub - scenarios in eight major medical scenarios.

Model Architecture

• It is a large language model with 7 billion parameters based on Transformer, using RoPE relative position encoding, SwiGLU activation function, and RMSNorm. The training uses Qwen - 7b¹ as the basic pre - trained model.

Main Characteristics

• High Accuracy: Trained on a large - scale medical corpus, it has high accuracy and low possibility of misdiagnosis.
• Scenario - oriented: It is specially optimized and customized for different medical scenarios and real needs to better serve application implementation.
• Iterative Optimization: It continuously collects and learns the latest medical research to improve model performance and system functions.

💻 Usage Examples

Basic Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers.generation import GenerationConfig

model_path = "WiNGPT2-7B-Chat"

tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_path, trust_remote_code=True)
model = model.eval()

generation_config = GenerationConfig(
  num_beams=1,
  top_p=0.75,
  top_k=30,
  repetition_penalty=1.1,
  max_new_tokens=1024
)

text = 'User: WiNGPT, 你好<|endoftext|>\n Assistant: '
inputs = tokenizer.encode(text, return_tensors="pt").to(device)
outputs = model.generate(inputs, generation_config=generation_config)
output = tokenizer.decode(outputs[0])
response = output.replace(inputs, '')

## 输出结果：你好！今天我能为你做些什么？<|endoftext|>

Advanced Usage

WiNGPT2 - 7B - Chat uses a custom prompt format:

User roles: User/Assistant

Prompt template: User: [space here]WiNGPT, 你好<|endoftext|>\n [space here]Assistant:; For multi - turn dialogue, splice according to this template, for example:

"User: WiNGPT, 你好<|endoftext|>\n Assistant:你好！今天我能为你做些什么？<|endoftext|>\n User: 你是谁？<|endoftext|>\n Assistant:"

It is recommended to use repetition_penalty = 1.1 [greedy search] when decoding.

Enterprise Service

13B model platform test (apply for the key directly)

📚 Documentation

Training Data

Data Overview

Medical Professional Data

Other Public Data

Continued Pre - training

• Expand the model's medical knowledge base: Pre - training data + part of the instruction data.

Instruction Fine - tuning

• Automatically construct a medical instruction set from data such as books, guidelines, cases, medical reports, and knowledge graphs.
• Manually annotate the instruction set. The data sources include electronic medical record systems, nursing medical record systems, PACS systems, clinical research systems, surgical management systems, public health scenarios, medical affairs management scenarios, and tool assistant scenarios.
• Use solutions such as FastChat², Self - Instruct³, and Evol - Instruct⁴ to expand the instruction set and enrich its diversified forms.

Data Engineering

• Data Classification: Classify according to the training stage and task scenario.
• Data Cleaning: Remove irrelevant information, correct spelling mistakes in the data, extract key information, and perform privacy - removing processing.
• Data Deduplication: Use the embedding method to remove duplicate data.
• Data Sampling: Perform targeted sampling according to the quality and distribution requirements of the dataset.

Model Card

Training Configuration and Parameters

Distributed Training Strategy and Parameters

• deepspeed + cpu_offload + zero_stage3
• gradient_checkpointing

Evaluation

Chinese Basic Model Evaluation C - EVAL (Zero - shot/Few - shot)

Chinese Medical Professional Evaluation MedQA - MCMLE (Zero - shot)

⚠️ Important Note

The current public evaluations have certain limitations, and the results are for reference only. More professional evaluations are coming soon.

🔧 Technical Details

WiNGPT2 is a large language model in the professional medical field. It can provide general users with anthropomorphic AI doctor consultations and Q&A functions, as well as knowledge Q&A in the general medical field. For professional medical personnel, the answers and suggestions provided by WiNGPT2 regarding patient diagnosis, medication, and health advice are for reference only.

You should understand that WiNGPT2 only provides information and suggestions and cannot replace the opinions, diagnoses, or treatment suggestions of medical professionals. Before using the information from WiNGPT2, please seek the advice of doctors or other medical professionals and independently evaluate the provided information.

The information in WiNGPT2 may be incorrect or inaccurate. Winning Health does not provide any express or implied guarantees regarding the accuracy, reliability, completeness, quality, safety, timeliness, performance, or applicability of WiNGPT2. You shall bear the results and decisions arising from the use of WiNGPT2. Winning Health shall not be responsible for any damages caused to you by third - party reasons.

📄 License

1. This project is licensed under the Apache License 2.0. The model weights need to comply with the relevant agreements and [license](https://github.com/QwenLM/Qwen - 7B/blob/main/LICENSE) of the base model [Qwen - 7B](https://github.com/QwenLM/Qwen - 7B). Please refer to its website for detailed content.
2. Please cite this project when using this project, including the model weights: https://github.com/winninghealth/WiNGPT2

🔗 References

1. https://github.com/QwenLM/Qwen - 7B
2. https://github.com/lm - sys/FastChat
3. https://github.com/yizhongw/self - instruct
4. https://github.com/nlpxucan/evol - instruct

📞 Contact Us

Website: https://www.winning.com.cn Email: wair@winning.com.cn