Biomistral 7B DARE

🚀 BioMistral-7B-mistral7instruct-dare

This project is a merged pre-trained language model created using mergekit. It combines the power of multiple models to offer enhanced performance in the medical and biological fields.

🚀 Quick Start

You can use BioMistral with Hugging Face's Transformers library as follows.

Basic Usage

from transformers import AutoModel, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("BioMistral/BioMistral-7B")
model = AutoModel.from_pretrained("BioMistral/BioMistral-7B")

✨ Features

• Model Merging: Utilizes the DARE TIES merge method to combine multiple pre-trained models.
• Multilingual Support: Supports multiple languages including English, French, Dutch, Spanish, Italian, Polish, Romanian, and German.
• Medical Domain Adaptation: Tailored for the medical and biological domains, pre-trained on PubMed Central data.

📦 Installation

The installation mainly involves using the transformers library. You can install it via the following command:

pip install transformers

📚 Documentation

Merge Details

Merge Method

This model was merged using the DARE TIES merge method, with mistralai/Mistral-7B-Instruct-v0.1 as the base model.

Models Merged

The following models were included in the merge:

• BioMistral/BioMistral-7B

Configuration

The following YAML configuration was used to produce this model:

models:
  - model: mistralai/Mistral-7B-Instruct-v0.1
    # No parameters necessary for base model
  - model: BioMistral/BioMistral-7B
    parameters:
      density: 0.5
      weight: 0.5
merge_method: dare_ties
base_model: mistralai/Mistral-7B-Instruct-v0.1
parameters:
  int8_mask: true
dtype: bfloat16

BioMistral Models

Quantized Models

Supervised Fine-tuning Benchmark

Supervised Fine-Tuning (SFT) performance of BioMistral 7B models compared to baselines, measured by accuracy (↑) and averaged across 3 random seeds of 3-shot. DARE, TIES, and SLERP are model merging strategies that combine BioMistral 7B and Mistral 7B Instruct. Best model in bold, and second-best underlined. *GPT-3.5 Turbo performances are reported from the 3-shot results without SFT.

🔧 Technical Details

BioMistral is an open-source LLM tailored for the biomedical domain, utilizing Mistral as its foundation model and further pre-trained on PubMed Central. The model merging process uses the DARE and TIES methods to combine the strengths of different pre-trained models.

📄 License

This project is licensed under the apache-2.0 license.

📖 Citation

Arxiv : https://arxiv.org/abs/2402.10373

@misc{labrak2024biomistral,
      title={BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains}, 
      author={Yanis Labrak and Adrien Bazoge and Emmanuel Morin and Pierre-Antoine Gourraud and Mickael Rouvier and Richard Dufour},
      year={2024},
      eprint={2402.10373},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

⚠️ Important Note

Although BioMistral is intended to encapsulate medical knowledge sourced from high-quality evidence, it hasn't been tailored to effectively, safely, or suitably convey this knowledge within professional parameters for action. We advise refraining from utilizing BioMistral in medical contexts unless it undergoes thorough alignment with specific use cases and undergoes further testing, notably including randomized controlled trials in real-world medical environments. BioMistral 7B may possess inherent risks and biases that have not yet been thoroughly assessed. Additionally, the model's performance has not been evaluated in real-world clinical settings. Consequently, we recommend using BioMistral 7B strictly as a research tool and advise against deploying it in production environments for natural language generation or any professional health and medical purposes.

⚠️ Important Note

Both direct and downstream users need to be informed about the risks, biases, and constraints inherent in the model. While the model can produce natural language text, our exploration of its capabilities and limitations is just beginning. In fields such as medicine, comprehending these limitations is crucial. Hence, we strongly advise against deploying this model for natural language generation in production or for professional tasks in the realm of health and medicine.