Rad Dino Maira 2

🚀 RAD-DINO-MAIRA-2

RAD-DINO-MAIRA-2 is a vision transformer model trained to encode chest X-rays using the self-supervised learning method DINOv2. It's mainly for research and can be used as a vision backbone for various downstream tasks.

🚀 Quick Start

from transformers import pipeline
pipe = pipeline(task="image-feature-extraction", model="microsoft/rad-dino-maira-2", pool=False)
patch_features = pipe("https://www.bhf.org.uk/-/media/images/information-support/tests/chest-x-ray/normal-chest-x-ray-620x400.jpg")

Refer to RAD-DINO for a more detailed example.

✨ Features

• Research - only Sharing: RAD - DINO - MAIRA - 2 is shared for research purposes only and is not meant for clinical practice.
• Versatile Downstream Tasks: It can be used as a vision backbone for tasks like image classification, segmentation, clustering, retrieval, and report generation.
• Trained on More Data: Compared to RAD - DINO, it was trained on more data.

📚 Documentation

Model description

RAD - DINO - MAIRA - 2 is a vision transformer model trained to encode chest X - rays using the self - supervised learning method DINOv2. It is a variant of [RAD - DINO](https://huggingface.co/microsoft/rad - dino), described in detail in RAD - DINO: Exploring Scalable Medical Image Encoders Beyond Text Supervision (F. Pérez - García, H. Sharma, S. Bond - Taylor, et al., 2024).

It is the version of RAD - DINO used in MAIRA - 2: Grounded Radiology Report Generation (S. Bannur, K. Bouzid, et al., 2024). Relative to [RAD - DINO](https://huggingface.co/microsoft/rad - dino), it was trained on more data.

• Developed by: Microsoft Health Futures
• Model type: Vision transformer
• License: MSRLA
• Finetuned from model: [dinov2 - base](https://huggingface.co/facebook/dinov2 - base)

Uses

RAD - DINO - MAIRA - 2 is shared for research purposes only. It is not meant to be used for clinical practice.

The model is a vision backbone that can be plugged to other models for downstream tasks. Some potential uses are:

• Image classification, with a classifier trained on top of the CLS token
• Image segmentation, with a decoder trained using the patch tokens
• Clustering, using the image embeddings directly
• Image retrieval, using nearest neighbors of the CLS token
• Report generation, with a language model to decode text

Fine - tuning RAD - DINO - MAIRA - 2 is typically not necessary to obtain good performance in downstream tasks.

Biases, risks, and limitations

RAD - DINO - MAIRA - 2 was trained with data from three countries, therefore it might be biased towards population in the training data. Underlying biases of the training datasets may not be well characterized.

Training details

Training data

We used images from five public and one private de - identified chest X - ray datasets to train RAD - DINO - MAIRA - 2.

Images in the validation and test sets used to train MAIRA - 2 were excluded from the training set of RAD - DINO - MAIRA - 2.

We used 8 nodes with 4 A100 GPUs each, and a batch size of 40 images per GPU. We share the last checkpoint, trained for 105,000 steps.

Training procedure

We refer to the manuscript for a detailed description of the training procedure.

Preprocessing

All DICOM files were resized using B - spline interpolation so that their shorter size was 518, min - max scaled to [0, 255], and stored as PNG files.

Training hyperparameters

• Training regime: fp16 using PyTorch - FSDP mixed - precision.

Evaluation

Our evaluation is best described in the manuscript.

Environmental impact

• Hardware type: NVIDIA A100 GPUs
• Hours used: 41 hours/GPU × 8 nodes × 4 GPUs/node = 1312 GPU - hours
• Cloud provider: Azure
• Compute region: West US 2
• Carbon emitted: 98.4 kg CO₂ eq.

Compute infrastructure

RAD - DINO - MAIRA - 2 was trained on [Azure Machine Learning](https://azure.microsoft.com/en - us/products/machine - learning).

Hardware

We used 8 Standard_NC96ads_A100_v4 nodes with four NVIDIA A100 (80 GB) GPUs each.

Software

We leveraged the code in DINOv2 for training. We used SimpleITK and Pydicom for processing of DICOM files.

Citation

BibTeX:

@misc{perezgarcia2024raddino,
      title={{RAD-DINO}: Exploring Scalable Medical Image Encoders Beyond Text Supervision},
      author={Fernando Pérez-García and Harshita Sharma and Sam Bond-Taylor and Kenza Bouzid and Valentina Salvatelli and Maximilian Ilse and Shruthi Bannur and Daniel C. Castro and Anton Schwaighofer and Matthew P. Lungren and Maria Wetscherek and Noel Codella and Stephanie L. Hyland and Javier Alvarez-Valle and Ozan Oktay},
      year={2024},
      eprint={2401.10815},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

APA:

Pérez - García, F., Sharma, H., Bond - Taylor, S., Bouzid, K., Salvatelli, V., Ilse, M., Bannur, S., Castro, D.C., Schwaighofer, A., Lungren, M.P., Wetscherek, M.T., Codella, N., Hyland, S.L., Alvarez - Valle, J., & Oktay, O. (2024). RAD - DINO: Exploring Scalable Medical Image Encoders Beyond Text Supervision. ArXiv, abs/2401.10815.

Model card contact

Fernando Pérez - García (fperezgarcia@microsoft.com).

📄 License

The model is released under the MSRLA license.