Vit Msn Small

🚀 Vision Transformer (small-sized model) pre-trained with MSN

A Vision Transformer (ViT) model pre-trained using the MSN method, offering effective feature extraction for image-related tasks.

🚀 Quick Start

The Vision Transformer (ViT) pre - trained with MSN is a powerful model for image - related tasks. It can extract useful features from images, which can be further used for downstream tasks like image classification.

✨ Features

• Joint - embedding architecture: MSN uses a joint - embedding architecture to match the prototypes of masked patches with that of the unmasked patches, achieving excellent performance in low - shot and extreme low - shot regimes.
• Effective pre - training: Through pre - training, the model learns an inner representation of images, which can be used to extract features for downstream tasks.
• Beneficial for small labeled datasets: The model is particularly useful when the training set has only a few labeled samples.

📦 Installation

To use this model, you need to install the transformers library. You can install it using pip:

pip install transformers

💻 Usage Examples

Basic Usage

Here is how to use this backbone encoder:

from transformers import AutoFeatureExtractor, ViTMSNModel
import torch
from PIL import Image
import requests

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

feature_extractor = AutoFeatureExtractor.from_pretrained("facebook/vit-msn-small")
model = ViTMSNModel.from_pretrained("facebook/vit-msn-small")
inputs = feature_extractor(images=image, return_tensors="pt")
with torch.no_grad():
    outputs = model(**inputs)
last_hidden_states = outputs.last_hidden_state

Advanced Usage

For fine - tuning on image classification use the ViTMSNForImageClassification class:

from transformers import AutoFeatureExtractor, ViTMSNForImageClassification
import torch
from PIL import Image
import requests

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

feature_extractor = AutoFeatureExtractor.from_pretrained("facebook/vit-msn-small")
model = ViTMSNForImageClassification.from_pretrained("facebook/vit-msn-small")

...

📚 Documentation

Model description

The Vision Transformer (ViT) is a transformer encoder model (BERT - like). Images are presented to the model as a sequence of fixed - size patches.

MSN presents a joint - embedding architecture to match the prototypes of masked patches with that of the unmasked patches. With this setup, their method yields excellent performance in the low - shot and extreme low - shot regimes.

By pre - training the model, it learns an inner representation of images that can then be used to extract features useful for downstream tasks: if you have a dataset of labeled images for instance, you can train a standard classifier by placing a linear layer on top of the pre - trained encoder.

Intended uses & limitations

You can use the raw model for downstream tasks like image classification. See the model hub to look for different versions of MSN pre - trained models that interest you. The model is particularly beneficial when you have a few labeled samples in your training set.

Citation

@article{assran2022masked,
  title={Masked Siamese Networks for Label - Efficient Learning}, 
  author={Assran, Mahmoud, and Caron, Mathilde, and Misra, Ishan, and Bojanowski, Piotr, and Bordes, Florian and Vincent, Pascal, and Joulin, Armand, and Rabbat, Michael, and Ballas, Nicolas},
  journal={arXiv preprint arXiv:2204.07141},
  year={2022}
}

📄 License

This model is released under the Apache 2.0 license.

📄 Information Table