Monet

🚀 MONET (Medical cONcept rETriever)

MONET is a vision - language foundation model. It can accurately annotate concepts in dermatology images and enables AI transparency in the development pipeline. It's trained on dermatological images paired with natural language descriptions from medical literature.

🚀 Quick Start

No specific quick - start steps are provided in the original document.

✨ Features

• Accurate Annotation: MONET can accurately annotate concepts across dermatology images, as verified by board - certified dermatologists, competing with supervised models built on previously concept - annotated dermatology datasets of clinical images.
• AI Transparency: It enables AI transparency across the entire AI system development pipeline, from building inherently interpretable models to dataset and model auditing.

📚 Documentation

Description

MONET is a CLIP ViT - L/14 vision - language foundation model trained on 105,550 dermatological images paired with natural language descriptions from a large collection of medical literature.

• Paper
• GitHub

Citation

@article{kim2024transparent,
    title={Transparent medical image AI via an image–text foundation model grounded in
medical literature},
    author={Chanwoo Kim and Soham U. Gadgil and Alex J. DeGrave and Jesutofunmi A. Omiye and Zhuo Ran Cai and Roxana Daneshjou and Su-In Lee},
    journal={Nature Medicine},
    year={2024},
    doi={10.1038/s41591-024-02887-x},
    url={https://doi.org/10.1038/s41591-024-02887-x}
}

Disclaimer: The model card is taken and modified from the official CLIP repository, it can be found here.

Model Details

The CLIP model was developed by researchers at OpenAI to learn about what contributes to robustness in computer vision tasks. The model was also developed to test the ability of models to generalize to arbitrary image classification tasks in a zero - shot manner. It was not developed for general model deployment - to deploy models like CLIP, researchers will first need to carefully study their capabilities in relation to the specific context they’re being deployed within.

Model Use

Intended Use

The model is intended as a research output for research communities. We hope that this model will enable researchers to better understand and explore zero - shot, arbitrary image classification. We also hope it can be used for interdisciplinary studies of the potential impact of such models - the CLIP paper includes a discussion of potential downstream impacts to provide an example for this sort of analysis.

Primary intended uses

The primary intended users of these models are AI researchers. We primarily imagine the model will be used by researchers to better understand robustness, generalization, and other capabilities, biases, and constraints of computer vision models.

Out - of - Scope Use Cases

⚠️ Important Note

Any deployed use case of the model - whether commercial or not - is currently out of scope. Non - deployed use cases such as image search in a constrained environment, are also not recommended unless there is thorough in - domain testing of the model with a specific, fixed class taxonomy. This is because our safety assessment demonstrated a high need for task specific testing especially given the variability of CLIP’s performance with different class taxonomies. This makes untested and unconstrained deployment of the model in any use case currently potentially harmful.

⚠️ Important Note

Certain use cases which would fall under the domain of surveillance and facial recognition are always out - of - scope regardless of performance of the model. This is because the use of artificial intelligence for tasks such as these can be premature currently given the lack of testing norms and checks to ensure its fair use.

⚠️ Important Note

Since the model has not been purposefully trained in or evaluated on any languages other than English, its use should be limited to English language use cases.