Vit Base Patch16 224.dino Mlxim

🚀 vit_base_patch16_224.dino

A Vision Transformer image classification model trained on ImageNet-1k dataset with DINO, offering self - supervised learning capabilities and ported to Apple MLX Framework.

This is a Vision Transformer image classification model. It was trained in a self - supervised manner on the ImageNet - 1k dataset. Only the backbone was trained, and no classification head was included. Note that this is a porting of the torch model weights to the Apple MLX Framework.

🚀 Quick Start

📦 Installation

pip install mlx-image

💻 Usage Examples

Basic Usage

Here is how to use this model for image classification:

from mlxim.model import create_model
from mlxim.io import read_rgb
from mlxim.transform import ImageNetTransform

transform = ImageNetTransform(train=False, img_size=224)
x = transform(read_rgb("cat.png"))
x = mx.expand_dims(x, 0)

model = create_model("vit_base_patch16_224.dino")
model.eval()

logits, attn_masks = model(x, attn_masks=True)

Advanced Usage

You can also use the embeds from the layer before the head:

from mlxim.model import create_model
from mlxim.io import read_rgb
from mlxim.transform import ImageNetTransform

transform = ImageNetTransform(train=False, img_size=512)
x = transform(read_rgb("cat.png"))
x = mx.expand_dims(x, 0)

# first option
model = create_model("vit_base_patch16_224.dino", num_classes=0)
model.eval()

embeds = model(x)

# second option
model = create_model("vit_base_patch16_224.dino")
model.eval()

embeds, attn_masks = model.get_features(x)

📚 Documentation

Attention maps

You can visualize the attention maps using the attn_masks returned by the model. Go check the mlx - image notebook.

📄 License

This project is licensed under the Apache - 2.0 license.