Mmed Llama 3 8B EnIns

🚀 MMedLM

The official model weights for "Towards Building Multilingual Language Model for Medicine".

🚀 Quick Start

The model can be loaded as follows:

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("Henrychur/MMed-Llama-3-8B-EnIns")
model = AutoModelForCausalLM.from_pretrained("Henrychur/MMed-Llama-3-8B-EnIns", torch_dtype=torch.float16)

• Inference format is similar to Llama 3-Instruct, you can check our inference code here.
• For multiple-choice question and answering tasks, we suggest using the following instruction.

from model import MedS_Llama3 # https://github.com/MAGIC-AI4Med/MedS-Ins/blob/main/Inference/model.py
sdk_api = MedS_Llama3(model_path="Henrychur/MMed-Llama-3-8B-EnIns", gpu_id=0)
INSTRUCTION = "Given a question and a list of options, select the correct answer from the options directly."
input_ = "Question: A mother brings her 3 - week - old infant to the pediatrician's office because she is concerned about his feeding habits. He was born without complications and has not had any medical problems up until this time. However, for the past 4 days, he has been fussy, is regurgitating all of his feeds, and his vomit is yellow in color. On physical exam, the child's abdomen is minimally distended but no other abnormalities are appreciated. Which of the following embryologic errors could account for this presentation?\nOptions: A: Abnormal migration of ventral pancreatic bud\tB: Complete failure of proximal duodenum to recanalize\tC: Abnormal hypertrophy of the pylorus\tD: Failure of lateral body folds to move ventrally and fuse in the midline\t"
results = sdk_api.chat([], input_, INSTRUCTION)
print(results)

✨ Features

This repo contains MMed-Llama 3-8B-EnIns, which is based on MMed-Llama 3-8B. We further fine-tune the model on English instruction fine-tuning dataset(from PMC-LLaMA). We did this for a fair comparison with existing models on commonly-used English benchmarks.

Notice that, MMed-Llama 3-8B-EnIns has only been trained on pmc_llama_instructions, which is an English medical SFT dataset focusing on QA tasks. So this model's ability to respond to multilingual input is still limited.

📚 Documentation

News

• [2024.2.21] Our pre-print paper is released on ArXiv. Dive into our findings here.
• [2024.2.20] We release MMedLM and MMedLM 2. With an auto-regressive continuous training on MMedC, these models achieve superior performance compared to all other open-source models, even rivaling GPT-4 on MMedBench.
• [2023.2.20] We release MMedC, a multilingual medical corpus containing 25.5B tokens.
• [2023.2.20] We release MMedBench, a new multilingual medical multi-choice question-answering benchmark with rationale. Check out the leaderboard here.

Evaluation on Commonly-used English Benchmark

The further pretrained MMed-Llama3 also showcases its great performance in the medical domain on different English benchmarks.

📄 License

The license for this project is llama3.

Contact

If you have any questions, please feel free to contact qiupengcheng@pjlab.org.cn.

Citation

@misc{qiu2024building,
      title={Towards Building Multilingual Language Model for Medicine}, 
      author={Pengcheng Qiu and Chaoyi Wu and Xiaoman Zhang and Weixiong Lin and Haicheng Wang and Ya Zhang and Yanfeng Wang and Weidi Xie},
      year={2024},
      eprint={2402.13963},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}