Wingpt2 14B Base

🚀 WiNGPT2

WiNGPT is a GPT-based large model tailored for the medical vertical domain. It aims to integrate professional medical knowledge, healthcare information, and data, offering intelligent information services such as medical Q&A, diagnostic support, and medical knowledge for the healthcare industry. This helps improve the efficiency of diagnosis and treatment and the quality of medical services.

🚀 Quick Start

WiNGPT (Winning Healthcare's large medical language model, hereinafter referred to as WiNGPT) commenced its R & D and training in January 2023.

In March, Winning Healthcare's AI laboratory completed the feasibility verification of WiNGPT - 001 and initiated internal testing. WiNGPT - 001 adopted a general GPT architecture with 6 billion parameters and achieved full - process self - development from pre - training to fine - tuning.

By May this year, the training data volume of WiNGPT - 001 reached 9,720 items of drug knowledge, 18 drug types, over 7,200 items of disease knowledge, over 2,800 items of inspection and testing knowledge, knowledge from 53 books, and more than 1,100 guideline documents. The total number of training tokens reached 3.7 billion.

In July, WiNGPT was upgraded to a 7B model with the latest model architecture, adding retrieval - enhanced generation capabilities. Meanwhile, the training of the 13B model began, and industry invitation testing was launched.

In September, WiNGPT underwent its latest version iteration, introducing the brand - new WiNGPT2. The new version can be easily extended and customized for various downstream application scenarios.

To give back to the open - source community, we open - sourced the WiNGPT2 - 7B/14B versions. Our intention is to accelerate the joint development of large medical language model technology and the industry through more open - source projects, ultimately benefiting human health.

✨ Features

Core Functions

• Medical Knowledge Q&A: It can answer questions related to medicine, health, diseases, etc., including but not limited to symptoms, treatments, drugs, prevention, and examinations.
• Natural Language Understanding: It can understand medical texts such as medical terms and medical records, providing key information extraction and categorization.
• Multi - turn Dialogue: It can play various medical professional roles, such as doctors, to have conversations with users and provide more accurate answers based on the context.
• Multi - task Support: It supports 32 medical tasks, covering 18 sub - scenarios in eight major medical scenarios.

Model Architecture

• It is a large language model based on Transformer with 7 billion/14 billion parameters. It uses RoPE relative position encoding, SwiGLU activation function, and RMSNorm. During training, Qwen - 7b¹ is used as the base pre - trained model.

Main Characteristics

• High Accuracy: Trained on a large - scale medical corpus, it has high accuracy and a low probability of misdiagnosis.
• Scenario - oriented: It is specifically optimized and customized for different medical scenarios and real - world needs, facilitating better application implementation.
• Iterative Optimization: It continuously collects and learns the latest medical research to improve model performance and system functionality.

💻 Usage Examples

Basic Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers.generation import GenerationConfig

model_path = "WiNGPT2-7B-Chat"

tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_path, trust_remote_code=True)
model = model.eval()

generation_config = GenerationConfig(
  num_beams=1,
  top_p=0.75,
  top_k=30,
  repetition_penalty=1.1,
  max_new_tokens=1024
)

text = 'User: WiNGPT, 你好<|endoftext|>\n Assistant: '
inputs = tokenizer.encode(text, return_tensors="pt").to(device)
outputs = model.generate(inputs, generation_config=generation_config)
output = tokenizer.decode(outputs[0])
response = output.replace(inputs, '')

## Output result: 你好！今天我能为你做些什么？<|endoftext|>

Advanced Usage

WiNGPT2 - 7B - Chat uses a custom prompt format:

User roles: User/Assistant

Prompt template: User: [space here]WiNGPT, 你好<|endoftext|>\n [space here]Assistant:; For multi - turn dialogues, splice according to this template, for example:

"User: WiNGPT, 你好<|endoftext|>\n Assistant:你好！今天我能为你做些什么？<|endoftext|>\n User: 你是谁？<|endoftext|>\n Assistant:"

It is recommended to use repetition_penalty = 1.1 [greedy search] when decoding.

Enterprise Service

13B model platform test (apply for the key directly)

📚 Documentation

Training Data

Data Overview

• Medical Professional Data | Property | Details | |----------|---------| | Drug Instructions | Knowledge Base | 15,000 items | | Multi - disease Knowledge Base | Knowledge Base | 9,720 items | | Medical Professional Books | Textbooks | 300 books | | Clinical Pathway Knowledge Base | Knowledge Base | 1,400 items | | Inspection and Testing Knowledge | Knowledge Base | 1.1 million items | | Multi - disciplinary Clinical Guidelines | Books | 1,100 copies from 18 departments | | Medical Knowledge Graph | Knowledge Base | 2.56 million triples | | Manually Annotated Datasets | Instructions | 50,000 items | | Medical Qualification Examination Questions | Test Questions | 300,000 items | | Medical Cases and Reports | Knowledge Base | 1 million items |

• Other Public Data | Property | Details | |----------|---------| | Medical Popular Science Books | Books | 500 books | | Other Multi - disciplinary Books | Books | 1,000 books | | Code | Instructions | 200,000 items | | General Test Questions | Test Questions | 3 million items | | Various Natural Language Processing Tasks | Instructions | 900,000 items | | Internet Texts | Internet | 3 million items | | Medical Q&A and Dialogues | Instructions | 5 million items |

Continued Pre - training

• Expand the model's medical knowledge base: Pre - training data + partial instruction data.

Instruction Fine - tuning

• Automatically construct a medical instruction set from data such as books, guidelines, cases, medical reports, and knowledge graphs.
• Manually annotate the instruction set. Data sources include electronic medical record systems, nursing medical record systems, PACS systems, clinical research systems, surgical management systems, public health scenarios, medical management scenarios, and tool assistant scenarios.
• Use schemes such as FastChat², Self - Instruct³, and Evol - Instruct⁴ to expand the instruction set and enrich its diverse forms.

Data Engineering

• Data Classification: Classify according to the training stage and task scenario.
• Data Cleaning: Remove irrelevant information, correct spelling errors in the data, extract key information, and perform de - privacy processing.
• Data Deduplication: Use the embedding method to remove duplicate data.
• Data Sampling: Perform targeted sampling according to the quality and distribution requirements of the dataset.

Model Card

Training Configuration and Parameters

Distributed Training Strategy and Parameters

• deepspeed + cpu_offload + zero_stage3
• gradient_checkpointing

Evaluation

Chinese Basic Model Evaluation C - EVAL (Zero - shot/Few - shot)

Chinese Medical Professional Evaluation MedQA - MCMLE (Zero - shot)

**Currently, public evaluations have certain limitations, and the results are for reference only. ** More professional evaluations are forthcoming.

Limitations and Disclaimer

(a) WiNGPT2 is a large language model in the professional medical field. It can provide general users with anthropomorphic AI doctor consultations and Q&A functions, as well as knowledge Q&A in the general medical field. For professional medical personnel, the answers and suggestions provided by WiNGPT2 regarding patient diagnosis, medication, and health advice are for reference only.

(b) You should understand that WiNGPT2 only provides information and suggestions and cannot replace the opinions, diagnoses, or treatment suggestions of medical professionals. Before using the information from WiNGPT2, please seek advice from doctors or other medical professionals and independently evaluate the provided information.

(c) The information in WiNGPT2 may contain errors or inaccuracies. Winning Healthcare does not provide any express or implied warranties regarding the accuracy, reliability, completeness, quality, safety, timeliness, performance, or applicability of WiNGPT2. You are solely responsible for the results and decisions made using WiNGPT2. Winning Healthcare shall not be liable for any damages caused by third - party reasons.

📄 License

1. This project is licensed under the Apache License 2.0. The model weights need to comply with the relevant agreements and [license](https://github.com/QwenLM/Qwen - 7B/blob/main/LICENSE) of the base model [Qwen - 7B](https://github.com/QwenLM/Qwen - 7B). Refer to its website for detailed information.
2. When using this project, including the model weights, please cite this project: https://github.com/winninghealth/WiNGPT2

References

1. https://github.com/QwenLM/Qwen - 7B
2. https://github.com/lm - sys/FastChat
3. https://github.com/yizhongw/self - instruct
4. https://github.com/nlpxucan/evol - instruct

Contact Us

Website: https://www.winning.com.cn Email: wair@winning.com.cn