license: other license_name: msrla license_link: https://huggingface.co/microsoft/rad-dino/blob/main/LICENSE library_name: transformers pipeline_tag: image-feature-extraction

Model card for RAD-DINO

Model description

RAD-DINO is a vision transformer model trained to encode chest X-rays using the self-supervised learning method DINOv2.

RAD-DINO is described in detail in Exploring Scalable Medical Image Encoders Beyond Text Supervision (F. Pérez-García, H. Sharma, S. Bond-Taylor, et al., 2024).

• Developed by: Microsoft Health Futures
• Model type: Vision transformer
• License: MSRLA
• Finetuned from model: dinov2-base

Uses

RAD-DINO is shared for research purposes only. It is not meant to be used for clinical practice.

The model is a vision backbone that can be plugged to other models for downstream tasks. Some potential uses are:

• Image classification, with a classifier trained on top of the CLS token
• Image segmentation, with a decoder trained using the patch tokens
• Clustering, using the image embeddings directly
• Image retrieval, using nearest neighbors of the CLS token
• Report generation, with a language model to decode text

Fine-tuning RAD-DINO is typically not necessary to obtain good performance in downstream tasks.

Biases, risks, and limitations

RAD-DINO was trained with data from three countries, therefore it might be biased towards population in the training data. Underlying biases of the training datasets may not be well characterized.

Getting started

Get some data

Let us first write an auxiliary function to download a chest X-ray.

>>> import requests
>>> from PIL import Image
>>> def download_sample_image() -> Image.Image:
...     """Download chest X-ray with CC license."""
...     base_url = "https://upload.wikimedia.org/wikipedia/commons"
...     image_url = f"{base_url}/2/20/Chest_X-ray_in_influenza_and_Haemophilus_influenzae.jpg"
...     headers = {"User-Agent": "RAD-DINO"}
...     response = requests.get(image_url, headers=headers, stream=True)
...     return Image.open(response.raw)
...

Load the model

Now let us download the model and encode an image.

>>> import torch
>>> from transformers import AutoModel
>>> from transformers import AutoImageProcessor
>>>
>>> # Download the model
>>> repo = "microsoft/rad-dino"
>>> rad_dino = AutoModel.from_pretrained(repo)
>>>
>>> # The processor takes a PIL image, performs resizing, center-cropping, and
>>> # intensity normalization using stats from MIMIC-CXR, and returns a
>>> # dictionary with a PyTorch tensor ready for the encoder
>>> processor = AutoImageProcessor.from_pretrained(repo)

Encode an image

>>> # Download and preprocess a chest X-ray
>>> image = download_sample_image()
>>> image.size  # (width, height)
(2765, 2505)
>>> inputs = processor(images=image, return_tensors="pt")
>>>
>>> # Encode the image!
>>> with torch.inference_mode():
>>>     outputs = rad_dino(**inputs)
>>>
>>> # Look at the CLS embeddings
>>> cls_embeddings = outputs.pooler_output
>>> cls_embeddings.shape  # (batch_size, num_channels)
torch.Size([1, 768])

If we are interested in the feature maps, we can reshape the patch embeddings into a grid. We will use einops (install with pip install einops) for this.

>>> def reshape_patch_embeddings(flat_tokens: torch.Tensor) -> torch.Tensor:
...     """Reshape flat list of patch tokens into a nice grid."""
...     from einops import rearrange
...     image_size = processor.crop_size["height"]
...     patch_size = model.config.patch_size
...     embeddings_size = image_size // patch_size
...     patches_grid = rearrange(flat_tokens, "b (h w) c -> b c h w", h=embeddings_size)
...     return patches_grid
...
>>> flat_patch_embeddings = outputs.last_hidden_state[:, 1:]  # first token is CLS
>>> reshaped_patch_embeddings = reshape_patch_embeddings(flat_patch_embeddings)
>>> reshaped_patch_embeddings.shape  # (batch_size, num_channels, height, width)
torch.Size([1, 768, 37, 37])

Weights for fine-tuning

We have released a checkpoint compatible with the original DINOv2 code to help researchers fine-tune our model.

First, let us write code to load a safetensors checkpoint.

>>> import safetensors
>>> def safetensors_to_state_dict(checkpoint_path: str) -> dict[str, torch.Tensor]:
...     state_dict = {}
...     with safe_open(checkpoint_path, framework="pt") as ckpt_file:
...         for key in ckpt_file.keys():
...             state_dict[key] = ckpt_file.get_tensor(key)
...     return state_dict
...

We can now use the hub model and load the RAD-DINO weights. Let's clone the DINOv2 repository so we can import the code for the head.

git clone https://github.com/facebookresearch/dinov2.git
cd dinov2

>>> import torch
>>> rad_dino_gh = torch.hub.load(".", "dinov2_vitb14")
>>> backbone_state_dict = safetensors_to_state_dict("backbone_compatible.safetensors")
>>> rad_dino_gh.load_state_dict(backbone_state_dict, strict=True)
<All keys matched successfully>

The weights of the head are also released:

>>> from dinov2.layers import DINOHead
>>> rad_dino_head_gh = DINOHead(
...    in_dim=768,
...    out_dim=65536,
...    hidden_dim=2048,
...    bottleneck_dim=256,
...    nlayers=3,
... )
>>> head_state_dict = safetensors_to_state_dict("dino_head.safetensors")
>>> rad_dino_head_gh.load_state_dict(head_state_dict, strict=True)
<All keys matched successfully>

Configs and augmentation

The configuration files ssl_default_config.yaml and vitb14_cxr.yaml, and the augmentations module are also available in the repository to help researchers reproduce the training procedure with our hyperparameters.

Training details

Training data

We used images from five public, deidentified chest X-ray datasets to train this checkpoint of RAD-DINO.

Images in the validation and test sets used to train MAIRA were excluded from the training set of RAD-DINO. The list of image files used for training is available at ./training_images.csv.

Note this checkpoint is different from the one in the paper, where some private data was used (and fewer GPUs). The checkpoint shared here is trained for 35 000 iterations (the total number of iterations in the run was 100 000, but we selected this checkpoint using linear probing on the validation sets of the evaluation datasets described in the paper). We used 16 nodes with 4 A100 GPUs each, and a batch size of 40 images per GPU.

Training procedure

We refer to the manuscript for a detailed description of the training procedure.

Preprocessing

All DICOM files were resized using B-spline interpolation so that their shorter size was 518, min-max scaled to [0, 255], and stored as PNG files.

Training hyperparameters

• Training regime: fp16 using PyTorch-FSDP mixed-precision.

Evaluation

Our evaluation is best described in the manuscript.

Environmental impact

• Hardware type: NVIDIA A100 GPUs
• Hours used: 40 hours/GPU × 16 nodes × 4 GPUs/node = 2560 GPU-hours
• Cloud provider: Azure
• Compute region: West US 2
• Carbon emitted: 222 kg CO₂ eq.

Compute infrastructure

RAD-DINO was trained on Azure Machine Learning.

Hardware

We used 16 Standard_NC96ads_A100_v4 nodes with four NVIDIA A100 (80 GB) GPUs each.

Software

We leveraged the code in DINOv2 for training. We used SimpleITK and Pydicom for processing of DICOM files.

Citation

BibTeX:

@article{perez-garcia_exploring_2025,
	title = {Exploring scalable medical image encoders beyond text supervision},
	issn = {2522-5839},
	url = {https://doi.org/10.1038/s42256-024-00965-w},
	doi = {10.1038/s42256-024-00965-w},
	journal = {Nature Machine Intelligence},
	author = {P{\'e}rez-Garc{\'i}a, Fernando and Sharma, Harshita and Bond-Taylor, Sam and Bouzid, Kenza and Salvatelli, Valentina and Ilse, Maximilian and Bannur, Shruthi and Castro, Daniel C. and Schwaighofer, Anton and Lungren, Matthew P. and Wetscherek, Maria Teodora and Codella, Noel and Hyland, Stephanie L. and Alvarez-Valle, Javier and Oktay, Ozan},
	month = jan,
	year = {2025},
}

APA:

Pérez-García, F., Sharma, H., Bond-Taylor, S., Bouzid, K., Salvatelli, V., Ilse, M., Bannur, S., Castro, D. C., Schwaighofer, A., Lungren, M. P., Wetscherek, M. T., Codella, N., Hyland, S. L., Alvarez-Valle, J., & Oktay, O. (2025). Exploring scalable medical image encoders beyond text supervision. In Nature Machine Intelligence. Springer Science and Business Media LLC. https://doi.org/10.1038/s42256-024-00965-w

Model card contact

Fernando Pérez-García (fperezgarcia@microsoft.com).