Merlin

🚀 Merlin: Vision Language Foundation Model for 3D Computed Tomography

Merlin is a 3D Vision Language Model (VLM) for computed tomography. It utilizes both structured electronic health records (EHR) and unstructured radiology reports for pretraining, offering a powerful solution for related fields.

Merlin is a 3D VLM for computed tomography that leverages both structured electronic health records (EHR) and unstructured radiology reports for pretraining. The huggingface repository here provides the model weights and an example image file.

[💻 Github] [📄 Paper]

🚀 Quick Start

For a quick start, you can install and use Merlin following the steps below.

✨ Features

• Leverage both structured EHR and unstructured radiology reports for pretraining.
• Provide model weights and an example image file in the Hugging Face repository.

📦 Installation

To install Merlin, you can simply run:

pip install merlin-vlm

For an editable installation, use the following commands to clone and install this repository.

git clone https://github.com/StanfordMIMI/Merlin.git
cd merlin
pip install -e .

For usage instructions, please visit the github repository.

📁 Project Structure:

.
├── README.md
├── i3_resnet_clinical_longformer_best_clip_04-02-2024_23-21-36_epoch_99.pt <Merlin weights>
├── image1.nii.gz <Sample Image>

📄 License

This project is licensed under the MIT license.

📎 Citation

If you find this repository useful for your work, please cite the original paper:

@article{blankemeier2024merlin,
  title={Merlin: A vision language foundation model for 3d computed tomography},
  author={Blankemeier, Louis and Cohen, Joseph Paul and Kumar, Ashwin and Van Veen, Dave and Gardezi, Syed Jamal Safdar and Paschali, Magdalini and Chen, Zhihong and Delbrouck, Jean-Benoit and Reis, Eduardo and Truyts, Cesar and others},
  journal={Research Square},
  pages={rs--3},
  year={2024}
}