Meditron 70b

🚀 Model Card for Meditron-70B-v1.0

Meditron is a suite of open - source medical Large Language Models (LLMs). Meditron-70B is adapted from Llama-2-70B for the medical domain and outperforms several models on multiple medical reasoning tasks.

🚀 Quick Start

Meditron-70B is now available for further testing and assessment. It can be used in various medical - related scenarios. For more interactive prompting, you can refer to our deployment guide which uses [FastChat](https://github.com/lm - sys/FastChat) with [vLLM](https://github.com/vllm - project/vllm).

✨ Features

• Medical - Domain Adapted: Adapted from Llama - 2 - 70B through continued pretraining on a curated medical corpus.
• High Performance: Outperforms Llama - 2 - 70B, GPT - 3.5 (text - davinci - 003, 8 - shot), and Flan - PaLM on multiple medical reasoning tasks.
• Diverse Use Cases: Suitable for medical exam question answering, differential diagnosis support, disease and general health information query.

📦 Installation

No installation steps are provided in the original document.

💻 Usage Examples

Basic Usage

It is possible to use this model to generate text for experimentation and understanding its capabilities. For example, you can use in - context learning with k demonstrations (3 or 5 as in the paper) added to the prompt.

Advanced Usage

For downstream question - answering tasks, you can finetune the model using specific training sets. You can also use a high - throughput and memory - efficient inference engine with a UI that supports chat and text generation.

📚 Documentation

Model Details

Uses

Direct Use

It can be used to generate text for experimentation, but not for production or work that may impact people.

Downstream Use

Meditron - 70B and Meditron - 7B are foundation models. They can be finetuned, instruction - tuned, or RLHF - tuned for specific downstream tasks. Two methods for downstream question - answering tasks are:

1. Apply in - context learning with k demonstrations added to the prompt.
2. Finetune the models using specific training sets.

Out - of - Scope Use

Do not use this model for natural language generation in a production environment, finetuned or otherwise.

Truthfulness, Helpfulness, Risk, and Bias

Truthfulness

An initial assessment of Meditron models' Truthfulness was done against baseline and consumer - level medical models using TruthfulQA (multiple choice). Only medical - relevant categories were focused on.

For more details, see the paper.

Helpfulness, Risk, and Bias

A comprehensive qualitative generation report of Meditron - 70B on medical - expert - designed queries is provided in the paper, comparing with Llama - 2 - 70B and ChatGPT - 3.5 (version Nov, 27, 2023).

Recommendations

⚠️ Important Note

Users should be aware of the risks, biases, and limitations of the model. Do not use it in production for natural language generation or professional health - related purposes without comprehensive testing.

Training Details

Training Data

Meditron’s domain - adaptive pre - training corpus GAP - Replay combines 48.1B tokens from four corpora:

• [Clinical Guidelines](https://huggingface.co/datasets/epfl - llm/guidelines): A new dataset of 46K internationally - recognized clinical practice guidelines.
• Medical Paper Abstracts: 16.1M abstracts from closed - access PubMed and PubMed Central papers.
• Medical Papers: Full - text articles from 5M publicly available PubMed and PubMed Central papers.
• Replay Data: 400M tokens of general domain data from [RedPajama - v1](https://huggingface.co/datasets/togethercomputer/RedPajama - Data - 1T)


Training Procedure

The [Megatron - LLM](https://github.com/epfLLM/Megatron - LLM) library was used for training. The hardware consists of 16 nodes of 8x NVIDIA A100 (80GB) SXM GPUs. A three - way parallelism scheme was used:

• Data Parallelism (DP) of 2
• Pipeline Parallelism (PP) of 8
• Tensor Parallelism (TP) of 8

Training Hyperparameters

Speeds, Sizes, Times

The model was trained in September and October 2023.

The 70B model was trained on 48e9 tokens at a throughput of about 40,200 tokens / second, with a bfloat16 model flops utilization of roughly 42.3%.

Evaluation

Testing Data & Metrics

Testing Data

• MedQA (USMLE)
• MedMCQA
• PubMedQA
• MMLU - Medical
• [MedQA - 4 - Option](https://huggingface.co/datasets/GBaker/MedQA - USMLE - 4 - options)

Metrics

• Accuracy: For multiple - choice question - answering tasks.

Results

Note: Models with * are already instruction - tuned and excluded from further finetuning.

Environmental Impact

• Hardware Type: 128 x NVIDIA A100 (80GB) SXM
• Total GPU hours: 42,496
• Hardware P (The original document seems incomplete here)

📄 License

The model is under the [LLAMA 2 COMMUNITY LICENSE AGREEMENT](https://huggingface.co/meta - llama/Llama - 2 - 70b/raw/main/LICENSE.txt), and the code is under APACHE 2.0 LICENSE.