Mmedlm

🚀 MMedLM

MMedLM is the official model weights for "Towards Building Multilingual Language Model for Medicine". It's a multilingual medical foundation model, aiming to enhance medical - domain knowledge across different languages. MMedLM 2 has been released, which is a more powerful multilingual medical foundation model, having undergone the same medical data enhancement pipeline as MMedLM.

💻Github Repo 🖨️arXiv Paper

🚀 Quick Start

The model can be loaded as follows:

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("Henrychur/MMedLM", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("Henrychur/MMedLM", torch_dtype=torch.float16, trust_remote_code=True)

⚠️ Important Note

• This is a foundation model that has not undergone instruction fine - tuning.
• Testing has found that using the latest version of transformers will result in errors. It is recommended to use transformers==4.28.1.

✨ Features

Model Architecture

This repo contains MMedLM, a multilingual medical foundation model with 7 billion parameters. MMedLM builds upon the foundation of InternLM and has been further pretrained on MMedC, a comprehensive multilingual medical corpus. This further pretraining enhances the model's medical - domain knowledge.

Pretraining Hyperparameters

The model underwent further pretraining on MMedC with the following hyperparameters:

• Iterations: 15000
• Global batch size: 512
• Cutoff length: 2048
• Learning rate: 2e - 5

📚 Documentation

News

• [2024.2.21] Our pre - print paper is released on ArXiv. Dive into our findings here.
• [2024.2.20] We release MMedLM and MMedLM 2. With an auto - regressive continues training on MMedC, these models achieve superior performance compared to all other open - source models, even rivaling GPT - 4 on MMedBench.
• [2023.2.20] We release MMedC, a multilingual medical corpus containing 25.5B tokens.
• [2023.2.20] We release MMedBench, a new multilingual medical multi - choice question - answering benchmark with rationale. Check out the leaderboard here.

Evaluation on MMedBench

The further pretrained MMedLM 2 showcases its great performance in the medical domain across different languages.

• GPT and Gemini are evaluated under zero - shot setting through API.
• Open - source models first undergo training on the trainset of MMedBench before evaluation.

📄 License

The model is released under the Apache 2.0 license.

Contact

If you have any questions, please feel free to contact qiupengcheng@pjlab.org.cn.

Citation

@misc{qiu2024building,
      title={Towards Building Multilingual Language Model for Medicine}, 
      author={Pengcheng Qiu and Chaoyi Wu and Xiaoman Zhang and Weixiong Lin and Haicheng Wang and Ya Zhang and Yanfeng Wang and Weidi Xie},
      year={2024},
      eprint={2402.13963},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}