Bio Medical MultiModal Llama 3 8B V1

🚀 Bio-Medical-MultiModal-Llama-3-8B-V1

Bio-Medical-MultiModal-Llama-3-8B-V1 is a specialized large language model designed for biomedical applications, fine - tuned from Llama - 3 - 8B - Instruct using a custom high - quality dataset.

🚀 Quick Start

This model is a fine - tuned Multimodal version of [meta - llama/Meta - Llama - 3 - 8B - Instruct](https://huggingface.co/meta - llama/Meta - Llama - 3 - 8B - Instruct) on our custom "BioMedData" text and image datasets.


✨ Features

• Specialized for biomedical applications, trained to understand and generate text related to various biomedical fields.
• Fine - tuned from Llama - 3 - 8B - Instruct using a diverse custom dataset with over 500,000 entries.
• Can handle multimodal data, including text and images, thanks to MiniCPM.

📦 Installation

The installation process is mainly about importing necessary libraries in Python. Here is the code to set up the environment:

import torch
from PIL import Image
from transformers import AutoModel, AutoTokenizer, BitsAndBytesConfig

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.float16,
)

💻 Usage Examples

Basic Usage

model = AutoModel.from_pretrained(
    "ContactDoctor/Bio - Medical - MultiModal - Llama - 3 - 8B - V1",
    quantization_config=bnb_config,
    device_map="auto",
    torch_dtype=torch.float16,
    trust_remote_code=True,
    attn_implementation="flash_attention_2",
)

tokenizer = AutoTokenizer.from_pretrained("ContactDoctor/Bio - Medical - MultiModal - Llama - 3 - 8B - V1", trust_remote_code=True)

image = Image.open("Path to Your image").convert('RGB')
question = 'Give the modality, organ, analysis, abnormalities (if any), treatment (if abnormalities are present)?'
msgs = [{'role': 'user', 'content': [image, question]}]

res = model.chat(
    image=image,
    msgs=msgs,
    tokenizer=tokenizer,
    sampling=True,
    temperature=0.95,
    stream=True
)

generated_text = ""
for new_text in res:
    generated_text += new_text
    print(new_text, flush=True, end='')

Expected Output

Sample Response

The modality is Magnetic Resonance Imaging (MRI), the organ being analyzed is the cervical spine, and there are no abnormalities present in the image.

📚 Documentation

Model details

Intended uses

Bio - Medical - MultiModal - Llama - 3 - 8B - V1 is intended for a wide range of applications within the biomedical field, including:

1. Research Support: Assisting researchers in literature review and data extraction from biomedical texts.
2. Clinical Decision Support: Providing information to support clinical decision - making processes.
3. Educational Tool: Serving as a resource for medical students and professionals seeking to expand their knowledge base.

Limitations and Ethical Considerations

Bio - Medical - MultiModal - Llama - 3 - 8B - V1 performs well in various biomedical NLP tasks, users should be aware of the following limitations:

1. Biases: The model may inherit biases present in the training data. Efforts have been made to curate a balanced dataset, but some biases may persist.
2. Accuracy: The model's responses are based on patterns in the data it has seen and may not always be accurate or up - to - date. Users should verify critical information from reliable sources.
3. Ethical Use: The model should be used responsibly, particularly in clinical settings where the stakes are high. It should complement, not replace, professional judgment and expertise.

Training and evaluation

Bio - Medical - MultiModal - Llama - 3 - 8B - V1 was trained using NVIDIA H100 GPU's, which provides the computational power necessary for handling large - scale data and model parameters efficiently. Rigorous evaluation protocols have been implemented to benchmark its performance against similar models, ensuring its robustness and reliability in real - world applications.

The model was trained using MiniCPM, which allowed us to efficiently handle the multimodal data. MiniCPM provided the ability to process and learn from visual information.

Contact Information

For further information, inquiries, or issues related to Biomed - LLM, please contact: Email: info@contactdoctor.in Website: https://www.contactdoctor.in

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 4
• eval_batch_size: 4
• Number of epochs: 3
• seed: 42
• gradient_accumulation_steps: 4
• optimizer: Adam with betas=(0.9, 0.999) and epsilon = 1e - 08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.03
• mixed_precision_training: Native AMP

Framework versions

• PEFT 0.11.0
• Transformers 4.40.2
• Pytorch 2.1.2
• Datasets 2.19.1
• Tokenizers 0.19.1

🔧 Technical Details

• The model is trained using NVIDIA H100 GPU's for efficient handling of large - scale data and parameters.
• MiniCPM is used to handle multimodal data, enabling the model to process and learn from visual information.

📄 License

This model is licensed under the Bio - Medical - MultiModal - Llama - 3 - 8B - V1 (Non - Commercial Use Only). Please review the terms and conditions before using the model.

Citation

If you use Bio - Medical - MultiModal - Llama - 3 - 8B - V1 in your research or applications, please cite it as follows:

@misc{ContactDoctor_MEDLLM,
  author = ContactDoctor,
  title = {Bio - Medical - MultiModal - Llama - 3 - 8B - V1: A High - Performance Biomedical Multimodal LLM},
  year = {2024},
  howpublished = {https://huggingface.co/ContactDoctor/Bio - Medical - MultiModal - Llama - 3 - 8B - V1},
}

Quick Demo