Clipmd

🚀 ClipMD: A Medical Image-Text Matching Model

ClipMD is a medical image - text matching model. It addresses the challenge of accurately matching medical images with relevant text, providing high - value assistance in medical information retrieval and diagnosis.

🚀 Quick Start

Use with Transformers

from PIL import Image

from transformers import AutoProcessor,AutoModel

model = AutoModel.from_pretrained("Idan0405/ClipMD",trust_remote_code=True)
processor = AutoProcessor.from_pretrained("Idan0405/ClipMD")

image = Image.open("your image path")

inputs = processor(text=["chest x-ray", "head MRI"], images=image, return_tensors="pt", padding=True)

outputs = model(**inputs)
logits_per_image = outputs[0] # this is the image-text similarity score
probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities

✨ Features

• ClipMD is based on OpenAI's CLIP model and utilizes a sliding window text encoder.
• It uses a ViT - B/32 Transformer architecture as an image encoder and a masked sliding window self - attention Transformer as a text encoder.
• The encoders are trained to maximize the similarity of (image, text) pairs via a contrastive loss.
• The model was fine - tuned on the ROCO dataset.

📚 Documentation

Model Details

Model Description

The model uses a ViT - B/32 Transformer architecture as an image encoder and a masked sliding window self - attention Transformer as a text encoder. These encoders are trained to maximize the similarity of (image, text) pairs via a contrastive loss. The model was fine - tuned on the ROCO dataset.

See also

• ClipMD repository on github.
• ClipMD paper on arxiv

Information Table