Meditron 7b

🚀 Meditron-7B-v1.0 Model Card

Meditron is a suite of open - source medical Large Language Models (LLMs). Meditron - 7B is a 7 - billion - parameter model adapted to the medical domain from Llama - 2 - 7B. It is trained on a curated medical corpus, including selected PubMed articles, abstracts, a new dataset of internationally - recognized medical guidelines, and general domain data from RedPajama - v1. Meditron - 7B, after finetuning on relevant training data, outperforms Llama - 2 - 7B and PMC - Llama on multiple medical reasoning tasks.

🚀 Quick Start

This section provides an overview of how to start using Meditron - 7B. For detailed deployment and usage, please refer to the relevant sections below.

✨ Features

• Medical Domain Adaptation: Adapted from Llama - 2 - 7B, Meditron - 7B is fine - tuned on a medical corpus, making it suitable for medical - related tasks.
• Multiple Use Cases: Can be used for medical exam question answering, differential diagnosis support, disease information query, and general health information query.
• Comparative Performance: Outperforms Llama - 2 - 7B and PMC - Llama on multiple medical reasoning tasks.

📦 Installation

The README does not provide specific installation steps, so this section is skipped.

💻 Usage Examples

Basic Usage

The README does not provide basic usage code examples, so this part is skipped.

Advanced Usage

The README does not provide advanced usage code examples, so this part is skipped.

📚 Documentation

Model Details

Model Sources

• Repository: epflLLM/meditron
• Trainer: epflLLM/Megatron - LLM
• Paper: MediTron - 70B: Scaling Medical Pretraining for Large Language Models

Uses

General Use Cases

Meditron - 7B is available for further testing and assessment as an AI assistant to enhance clinical decision - making and provide access to an LLM for healthcare use. Potential use cases include:

• Medical exam question answering
• Supporting differential diagnosis
• Disease information (symptoms, cause, treatment) query
• General health information query

Direct Use

It can be used to generate text for experimentation and understanding its capabilities. However, it should not be used directly for production or work that may impact people.

Downstream Use

Meditron - 70B and Meditron - 7B are foundation models without finetuning or instruction - tuning. They can be finetuned, instruction - tuned, or RLHF - tuned for specific downstream tasks and applications. Two methods are used for downstream question - answering tasks:

1. Apply in - context learning with k demonstrations (3 or 5 in the paper) added to the prompt.
2. Finetune the models for downstream question - answering tasks using specific training sets.

We encourage the adaptation of the base model for more diverse applications. For a more interactive way to prompt the model, we recommend using a high - throughput and memory - efficient inference engine with a UI that supports chat and text generation. You can check out our deployment guide, where we used [FastChat](https://github.com/lm - sys/FastChat) with [vLLM](https://github.com/vllm - project/vllm). We collected generations for our qualitative analysis through an interactive UI platform, BetterChatGPT. Here is an example of the prompt format we used:

Out - of - Scope Use

We do not recommend using this model for natural language generation in a production environment, whether finetuned or not.

Truthfulness, Helpfulness, Risk, and Bias

We conducted an initial assessment of Meditron models' Truthfulness against baseline models and consumer - level medical models. We used TruthfulQA (multiple choice) as the main evaluation benchmark, focusing on medical - relevant categories such as Health, Nutrition, Psychology, and Science. For 7B models, we performed one - shot evaluations for consistent answer generation, and for 70B models, the evaluations were under the zero - shot setting.

For a more detailed performance analysis, please refer to the paper. Significant research is still needed to fully explore potential bias, fairness, and safety issues with this language model. Our evaluation of Meditron - 7B's helpfulness, risk, and bias is highly limited. Therefore, as noted in the safety notice, we strongly oppose any deployment in medical applications without further alignment and rigorous evaluation.

Recommendations

IMPORTANT! Users (both direct and downstream) should be aware of the risks, biases, and limitations of the model. Although the model can generate natural language text, our exploration of its capabilities and limitations is still in the early stage. Understanding these limitations is particularly important in the medical domain. Therefore, we strongly recommend against using this model in production for natural language generation or for professional purposes related to health and medicine.

Training Details

Training Data

Meditron’s domain - adaptive pre - training corpus GAP - Replay combines 48.1B tokens from four corpora:

• Clinical Guidelines: A new dataset of 46K internationally - recognized clinical practice guidelines from various healthcare - related sources, including hospitals and international organizations.
• Medical Paper Abstracts: 16.1M abstracts extracted from closed - access PubMed and PubMed Central papers.
• Medical Papers: Full - text articles extracted from 5M publicly available PubMed and PubMed Central papers.
• Replay Data: 400M tokens of general domain pretraining data sampled from RedPajama - v1

Data Preprocessing

Please refer to the paper for detailed preprocessing procedures.

Training Procedure

We used the Megatron - LLM distributed training library, a derivative of Nvidia's Megatron LM project, to optimize training efficiency. The hardware consists of 1 node of 8x NVIDIA A100 (80GB) SXM GPUs connected by NVLink and NVSwitch, with a single Nvidia ConnectX - 6 DX network card, and is equipped with 2 x AMD EPYC 7543 32 - Core Processors and 512 GB of RAM.

Our three - way parallelism scheme uses:

• Data Parallelism (DP -- different GPUs process different subsets of the batches) of 2.
• Pipeline Parallelism (PP -- different GPUs process different layers) of 4.
• Tensor Parallelism (TP -- different GPUs process different subtensors for matrix multiplication) of 1.

Training Hyperparameters

Sizes

The model was trained in September 2023. The model architecture is exactly Llama 2, with the following specifications:

Evaluation

Testing Data & Metrics

Testing Data

• MedQA (USMLE)
• MedMCQA
• PubMedQA
• MMLU - Medical
• MedQA - 4 - Option

Metrics

• Accuracy: Suitable for evaluating multiple - choice question - answering tasks.

Results

We finetuned meditron - 7b, llama - 2 - 7b, pmc - llama - 7b on each benchmark (pubmedqa, medmcqa, medqa)'s training data individually. We report the finetuned models' performance with top token selection as the inference mode. For MMLU - Medical, models finetuned on MedMCQA are used for inference. For MedQA - 4 - Option, models finetuned on MedQA are used for inference.

Note: Models with * are already instruction - tuned, so they are excluded from further finetuning on any training data.

Environmental Impact

• Hardware Type: 8 x NVIDIA A100 (80GB) SXM
• Total GPU hours: 588.8
• Hardware Provider: EPFL Research Computing Platform
• Compute Region: Switzerland
• Carbon Emitted: Switzerland has a carbon efficiency of 0.016 kgCO2/kWh (https://www.carbonfootprint.com/docs/2018_8_electricity_factors_august_2018_-_online_sources.pdf). 73.6 hours of 8 A100s means 588.8 hours at a TDP of 400W. Assuming a Power Usage effectiveness of 1.5, the total emissions are estimated to be: (400W / 1000W/kWh / GPU * 0.016 kgCO2/kWh * 73.6 h * 8 GPU) * 1.8 PUE = 6.8 kgCO2.

Citation

If you use Meditron or its training data, please cite our work:

@misc{chen2023meditron70b,
      title={MEDITRON-70B: Scaling Medical Pretraining for Large Language Models}, 
      author={Zeming Chen and Alejandro Hernández-Cano and Angelika Romanou and Antoine Bonnet and Kyle Matoba and Francesco Salvi and Matteo Pagliardini and Simin Fan and Andreas Köpf and Amirkeivan Mohtashami and Alexandre Sallinen and Alireza Sakhaeirad and Vinitra Swamy and Igor Krawczuk and Deniz Bayazit and Axel Marmet and Syrielle Montariol and Mary-Anne Hartley and Martin Jaggi and Antoine Bosselut},
      year={2023},
      eprint={2311.16079},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

@software{epfmedtrn,
  author = {Zeming Chen and Alejandro Hernández-Cano and Angelika Romanou and Antoine Bonnet and Kyle Matoba and Francesco Salvi and Matteo Pagliardini and Simin Fan and Andreas Köpf and Amirkeivan Mohtashami and Alexandre Sallinen and Alireza Sakhaeirad and Vinitra Swamy and Igor Krawczuk and Deniz Bayazit and Axel Marmet and Syrielle Montariol and Mary-Anne Hartley and Martin Jaggi and Antoine Bosselut},
  title = {MediTron-70B: Scaling Medical Pretraining for Large Language Models},
  month = November,
  year = 2023,
  url = {https://github.com/epfLLM/meditron}
}