Data2vec Vision Large

🚀 Data2Vec-Vision (large-sized model, pre-trained only)

A BEiT model pre-trained in a self-supervised manner on ImageNet-1k, offering potential for image classification tasks.

🚀 Quick Start

The Data2Vec-Vision large-sized pre-trained model is a BEiT model pre-trained in a self-supervised fashion on ImageNet-1k (1.2 million images, 1000 classes) at a resolution of 224x224. It was introduced in the paper data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language by Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, Michael Auli and first released in this repository.

Disclaimer: The Facebook team that released this model did not write a model card for it. This model card has been written by the Hugging Face team.

✨ Features

Pre-Training method

For more information, please refer to the official paper.

Abstract

While the general idea of self-supervised learning is identical across modalities, the actual algorithms and objectives differ widely because they were developed with a single modality in mind. To get us closer to general self-supervised learning, we present data2vec, a framework that uses the same learning method for either speech, NLP or computer vision. The core idea is to predict latent representations of the full input data based on a masked view of the input in a self-distillation setup using a standard Transformer architecture. Instead of predicting modality-specific targets such as words, visual tokens or units of human speech which are local in nature, data2vec predicts contextualized latent representations that contain information from the entire input. Experiments on the major benchmarks of speech recognition, image classification, and natural language understanding demonstrate a new state of the art or competitive performance to predominant approaches.

Intended uses & limitations

You can utilize the raw model for image classification. Check the model hub to find fine-tuned versions for tasks that interest you.

📚 Documentation

Training data

The BEiT model was pre-trained on ImageNet-1k, a dataset comprising 1.2 million images and 1000 classes.

Training procedure

Preprocessing

The precise details of image preprocessing during training/validation can be found here.

Images are resized/rescaled to the same resolution (224x224) and normalized across the RGB channels with a mean of (0.5, 0.5, 0.5) and a standard deviation of (0.5, 0.5, 0.5).

Pretraining

For all pre-training related hyperparameters, please refer to the original paper and the original codebase.

Evaluation results

For evaluation results on several image classification benchmarks, refer to Table 1 of the original paper. Note that for fine-tuning, the best results are achieved with a higher resolution. Naturally, increasing the model size will lead to better performance.

BibTeX entry and citation info

@misc{https://doi.org/10.48550/arxiv.2202.03555,
  doi = {10.48550/ARXIV.2202.03555},
  url = {https://arxiv.org/abs/2202.03555},
  author = {Baevski, Alexei and Hsu, Wei-Ning and Xu, Qiantong and Babu, Arun and Gu, Jiatao and Auli, Michael},
  keywords = {Machine Learning (cs.LG), FOS: Computer and information sciences, FOS: Computer and information sciences},
  title = {data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language},
  publisher = {arXiv},
  year = {2022},
  copyright = {arXiv.org perpetual, non-exclusive license}
}

📄 License

This model is licensed under the Apache-2.0 license.