Mmed Llama 3 8B

🚀 MMedLM

The official model weights for "Towards Building Multilingual Language Model for Medicine".

💻Github Repo 🖨️arXiv Paper

🚀 Quick Start

The model can be loaded as follows:

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("Henrychur/MMed-Llama-3-8B")
model = AutoModelForCausalLM.from_pretrained("Henrychur/MMed-Llama-3-8B", torch_dtype=torch.float16)

⚠️ Important Note

This is a foundation model that has not undergone instruction fine-tuning.

✨ Features

This repo contains MMed-Llama 3, a multilingual medical foundation model with 8 billion parameters. MMed-Llama 3 builds upon the foundation of Llama 3 and has been further pretrained on MMedC, a comprehensive multilingual medical corpus. This further pretraining enhances the model's medical-domain knowledge.

The model underwent further pretraining on MMedC with the following hyperparameters:

• Iterations: 15000
• Global batch size: 512
• Cutoff length: 8192
• Learning rate: 2e-5

📚 Documentation

News

• [2024.2.21] Our pre-print paper is released on ArXiv. Dive into our findings here.
• [2024.2.20] We release MMedLM and MMedLM 2. With an auto-regressive continues training on MMedC, these models achieves superior performance compared to all other open-source models, even rivaling GPT-4 on MMedBench.
• [2023.2.20] We release MMedC, a multilingual medical corpus containing 25.5B tokens.
• [2023.2.20] We release MMedBench, a new multilingual medical multi-choice question-answering benchmark with rationale. Check out the leaderboard here.

Evaluation on MMedBench

The further pretrained MMedLM 2 showcases its great performance in the medical domain across different languages.

• GPT and Gemini are evaluated under zero-shot setting through API.
• Open-source models first undergo training on the trainset of MMedBench before evaluation.

📄 License

The model uses the llama3 license.

📞 Contact

If you have any questions, please feel free to contact qiupengcheng@pjlab.org.cn.

📚 Citation

@misc{qiu2024building,
      title={Towards Building Multilingual Language Model for Medicine}, 
      author={Pengcheng Qiu and Chaoyi Wu and Xiaoman Zhang and Weixiong Lin and Haicheng Wang and Ya Zhang and Yanfeng Wang and Weidi Xie},
      year={2024},
      eprint={2402.13963},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}