Dinov2 With Registers Base

🚀 Vision Transformer (base-sized model) trained using DINOv2, with registers

This is a Vision Transformer (ViT) model that can extract image features. It was first introduced in the paper Vision Transformers Need Registers and released in this repository.

📦 Information Table

🚀 Quick Start

The Vision Transformer (ViT) model was initially introduced for supervised image classification on ImageNet. Subsequently, methods were developed to enable it to perform effectively in self - supervised image feature extraction without the need for labels.

✨ Features

• Artifact - free: By adding register tokens during pre - training, the model eliminates artifacts in attention maps.
• Interpretable attention maps: The attention maps become more interpretable.
• Improved performance: It shows enhanced performance in image feature extraction.

📚 Documentation

Model description

The Vision Transformer (ViT) is a transformer encoder model (similar to BERT) initially introduced for supervised image classification on ImageNet. Later, it was adapted for self - supervised image feature extraction.

The authors of DINOv2 found that ViTs had artifacts in attention maps due to using some image patches as “registers”. They proposed adding new “register” tokens only during pre - training, which led to artifact - free attention maps, interpretable attention maps, and improved performance.

Visualization of attention maps of various models trained with vs. without registers. Taken from the original paper.

Note that this model does not include any fine - tuned heads. Through pre - training, the model learns an inner representation of images, which can be used for downstream tasks. For example, a linear layer can be placed on top of the pre - trained encoder to train a standard classifier.

Intended uses & limitations

You can use the raw model for feature extraction. Check the model hub for fine - tuned versions for specific tasks.

💻 Usage Examples

Basic Usage

from transformers import AutoImageProcessor, AutoModel
from PIL import Image
import requests

url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
image = Image.open(requests.get(url, stream=True).raw)

processor = AutoImageProcessor.from_pretrained('facebook/dinov2-with-registers-base')
model = AutoModel.from_pretrained('facebook/dinov2-with-registers-base')

inputs = processor(images=image, return_tensors="pt")
outputs = model(**inputs)
last_hidden_states = outputs.last_hidden_state

📄 License

This model is licensed under the apache - 2.0 license.

BibTeX entry and citation info

@misc{darcet2024visiontransformersneedregisters,
      title={Vision Transformers Need Registers}, 
      author={Timothée Darcet and Maxime Oquab and Julien Mairal and Piotr Bojanowski},
      year={2024},
      eprint={2309.16588},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2309.16588}, 
}