Dinov2 With Registers Small Imagenet1k 1 Layer

🚀 Vision Transformer (small-sized model) trained using DINOv2, with registers

A Vision Transformer (ViT) model trained with DINOv2 and registers for image classification.

🚀 Quick Start

You can use the raw model to classify an image into one of the 1000 possible ImageNet classes. See the model hub to look for fine - tuned versions on a task that interests you.

Here is how to use this model:

from transformers import AutoImageProcessor, AutoModelForImageClassification
import torch
from PIL import Image
import requests

url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
image = Image.open(requests.get(url, stream=True).raw)

processor = AutoImageProcessor.from_pretrained('facebook/dinov2-with-registers-small-imagenet1k-1-layer')
model = AutoModelForImageClassification.from_pretrained('facebook/dinov2-with-registers-small-imagenet1k-1-layer')

inputs = processor(images=image, return_tensors="pt")

with torch.no_grad():
  outputs = model(**inputs)
  logits = outputs.logits

class_idx = outputs.logits.argmax(-1).item()
print("Predicted class:", model.config.id2label[class_idx])

✨ Features

• Artifact - free: By adding "register" tokens during pre - training, the model has no artifacts in attention maps.
• Interpretable attention maps: The use of register tokens makes the attention maps interpretable.
• Improved performance: The model shows improved performance in image feature extraction and classification tasks.

📚 Documentation

Model description

The Vision Transformer (ViT) is a transformer encoder model (BERT - like) originally introduced to do supervised image classification on ImageNet.

Next, people figured out ways to make ViT work really well on self - supervised image feature extraction (i.e. learning meaningful features, also called embeddings) on images without requiring any labels. Some example papers here include DINOv2 and MAE.

The authors of DINOv2 noticed that ViTs have artifacts in attention maps. It’s due to the model using some image patches as “registers”. The authors propose a fix: just add some new tokens (called "register" tokens), which you only use during pre - training (and throw away afterwards). This results in:

• no artifacts
• interpretable attention maps
• and improved performances.

Visualization of attention maps of various models trained with vs. without registers. Taken from the original paper.

Note that this model does not include any fine - tuned heads.

By pre - training the model, it learns an inner representation of images that can then be used to extract features useful for downstream tasks: if you have a dataset of labeled images for instance, you can train a standard classifier by placing a linear layer on top of the pre - trained encoder. One typically places a linear layer on top of the [CLS] token, as the last hidden state of this token can be seen as a representation of an entire image.

BibTeX entry and citation info

@misc{darcet2024visiontransformersneedregisters,
      title={Vision Transformers Need Registers}, 
      author={Timothée Darcet and Maxime Oquab and Julien Mairal and Piotr Bojanowski},
      year={2024},
      eprint={2309.16588},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2309.16588}, 
}

📄 License

This model is released under the apache - 2.0 license.