MELT TinyLlama 1.1B Chat V1.0

🚀 Model Card MELT-TinyLlama-1.1B-Chat-v1.0

The MELT-TinyLlama-1.1B-Chat-v1.0 Large Language Model (LLM) is a pretrained generative text model. It has been pre-trained and fine-tuned using publicly available medical data. This model shows a 13.76% improvement over TinyLlama-1.1B-Chat-v1.0 across 3 medical benchmarks, including examples from USMLE, Indian AIIMS, and NEET medical examinations.

✨ Features

• Trained on a wide range of medical domain data, including text, chat, Q/A, and instruction data.
• Demonstrates significant improvement over its base model in medical benchmarks.
• Intended for research purposes, especially suitable for QA or chat format prompts.

📚 Documentation

Model Details

The Medical Education Language Transformer (MELT) models have been trained on a wide - range of text, chat, Q/A, and instruction data in the medical domain. Although the model was evaluated using publicly available USMLE, Indian AIIMS, and NEET medical examination example questions, its application is intended to be more broadly applicable.

Model Description

• Developed by: Center for Applied AI
• Funded by: Institute or Biomedical Informatics
• Model type: LLM
• Language(s) (NLP): English
• License: Apache 2.0
• Finetuned from model: TinyLlama-1.1B-Chat-v1.0

Uses

MELT is intended for research purposes only. MELT models are best suited for prompts using a QA or chat format.

Out-of-Scope Use

MELT is intended for research purposes only and should not be used for medical advice.

Bias, Risks, and Limitations

MELT was trained using publicly available data collections, which likely contain biased and inaccurate information. The training and evaluation datasets have not been evaluated for content or accuracy.

How to Get Started with the Model

Use this model like you would any llama - 2 - 7b - chat - hf model.

Training Details

Training Data

The following datasets were used for training:

• Expert Med
• MedQA train
• MedMCQA train
• LiveQA
• MedicationQA
• MMLU clinical topics
• Medical Flashcards
• Wikidoc
• Wikidoc Patient Information
• MEDIQA
• MMMLU
• icliniq 10k
• HealthCare Magic 100k
• GenMedGPT-5k
• Mental Health Conversational

Training Procedure

Training Hyperparameters

• Lora Rank: 64
• Lora Alpha: 16
• Lora Targets: "o_proj","down_proj","v_proj","gate_proj","up_proj","k_proj","q_proj"
• LR: 2e - 4
• Epoch: 3
• Precision: bf16

Evaluation

MELT-TinyLlama-1.1B-Chat-v1.0 demonstrates an average 13.76% improvement over TinyLlama-1.1B-Chat-v1.0 across 3 USMLE, Indian AIIMS, and NEET medical examination benchmarks.

TinyLlama-1.1B-Chat-v1.0

• medqa: {'base': {'Average': 25.49, 'STEP-1': 24.48, 'STEP-2&3': 26.64}}
• mausmle: {'base': {'Average': 19.71, 'STEP-1': 21.18, 'STEP-2': 20.69, 'STEP-3': 17.76}}
• medmcqa: {'base': {'Average': 28.52, 'MEDICINE': 29.35, 'OPHTHALMOLOGY': 28.57, 'ANATOMY': 30.82, 'PATHOLOGY': 29.07, 'PHYSIOLOGY': 20.45, 'DENTAL': 30.09, 'RADIOLOGY': 14.29, 'BIOCHEMISTRY': 22.31, 'ANAESTHESIA': 26.09, 'GYNAECOLOGY': 24.84, 'PHARMACOLOGY': 32.02, 'SOCIAL': 31.11, 'PEDIATRICS': 31.82, 'ENT': 28.95, 'SURGERY': 31.45, 'MICROBIOLOGY': 26.03, 'FORENSIC': 16.28, 'PSYCHIATRY': 22.22, 'SKIN': 40.0, 'ORTHOPAEDICS': 21.43, 'UNKNOWN': 0.0}}
• average: 24.57%

MELT-TinyLlama-1.1B-Chat-v1.0

• medqa: {'base': {'Average': 29.5, 'STEP-1': 28.17, 'STEP-2&3': 31.03}}
• mausmle: {'base': {'Average': 21.51, 'STEP-1': 27.06, 'STEP-2': 19.54, 'STEP-3': 18.69}}
• medmcqa: {'base': {'Average': 32.84, 'MEDICINE': 27.72, 'OPHTHALMOLOGY': 38.1, 'ANATOMY': 39.73, 'PATHOLOGY': 32.56, 'PHYSIOLOGY': 35.61, 'DENTAL': 32.23, 'RADIOLOGY': 41.07, 'BIOCHEMISTRY': 33.06, 'ANAESTHESIA': 39.13, 'GYNAECOLOGY': 22.88, 'PHARMACOLOGY': 32.58, 'SOCIAL': 26.67, 'PEDIATRICS': 34.09, 'ENT': 42.11, 'SURGERY': 33.47, 'MICROBIOLOGY': 30.14, 'FORENSIC': 41.86, 'PSYCHIATRY': 55.56, 'SKIN': 60.0, 'ORTHOPAEDICS': 35.71, 'UNKNOWN': 100.0}}
• average: 27.95%

Testing Data

• MedQA test
• MedMCQA test
• MA USMLE

Disclaimer

The use of large language models, such as this one, is provided without warranties or guarantees of any kind. While every effort has been made to ensure accuracy, completeness, and reliability of the information generated, it should be noted that these models may produce responses that are inaccurate, outdated, or inappropriate for specific purposes. Users are advised to exercise discretion and judgment when relying on the information generated by these models. The outputs should not be considered as professional, legal, medical, financial, or any other form of advice. It is recommended to seek expert advice or consult appropriate sources for specific queries or critical decision - making. The creators, developers, and providers of these models disclaim any liability for damages, losses, or any consequences arising from the use, reliance upon, or interpretation of the information provided by these models. The user assumes full responsibility for their interactions and usage of the generated content. By using these language models, users agree to indemnify and hold harmless the developers, providers, and affiliates from any claims, damages, or liabilities that may arise from their use. Please be aware that these models are constantly evolving, and their capabilities, limitations, and outputs may change over time without prior notice. Your use of this language model signifies your acceptance and understanding of this disclaimer.

📄 License

This model is licensed under the Apache 2.0 license.