Beit Base Patch16 224 Pt22k Ft22k

🚀 BEiT (base-sized model, fine-tuned on ImageNet-22k)

BEiT is a model pre - trained in a self - supervised manner on ImageNet - 22k (also known as ImageNet - 21k, with 14 million images and 21,841 classes) at a resolution of 224x224. It is then fine - tuned on the same dataset at the same resolution. This model was introduced in the paper BEIT: BERT Pre - Training of Image Transformers by Hangbo Bao, Li Dong, and Furu Wei, and was first released in this repository.

Disclaimer: The team releasing BEiT did not write a model card for this model. This model card is written by the Hugging Face team.

🚀 Quick Start

The BEiT model can be used for image classification tasks. You can find fine - tuned versions on the model hub for specific tasks that interest you.

✨ Features

• Self - supervised pre - training: BEiT is pre - trained on a large collection of images (ImageNet - 21k) in a self - supervised fashion, which helps it learn an inner representation of images.
• Relative position embeddings: Unlike the original ViT models, BEiT uses relative position embeddings (similar to T5), which can better capture the relative spatial relationships between patches.
• Mean - pooling for classification: BEiT performs image classification by mean - pooling the final hidden states of the patches, rather than using a linear layer on top of the [CLS] token's final hidden state.

📚 Documentation

Model description

The BEiT model is a Vision Transformer (ViT), a transformer encoder model (BERT - like). Different from the original ViT model, BEiT is pre - trained on ImageNet - 21k at a resolution of 224x224 pixels in a self - supervised way. The pre - training objective is to predict visual tokens from the encoder of OpenAI's DALL - E's VQ - VAE based on masked patches.

After pre - training, the model is fine - tuned in a supervised manner on ImageNet (ILSVRC2012), a dataset with 1 million images and 1,000 classes, also at a resolution of 224x224.

Images are presented to the model as a sequence of fixed - size patches (resolution 16x16), which are linearly embedded. BEiT uses relative position embeddings instead of absolute position embeddings and classifies images by mean - pooling the final hidden states of the patches.

By pre - training, the model learns an inner representation of images that can be used to extract features for downstream tasks. For example, if you have a labeled image dataset, you can train a standard classifier by placing a linear layer on top of the pre - trained encoder.

Intended uses & limitations

You can use the raw model for image classification. Check the model hub for fine - tuned versions for specific tasks.

💻 Usage Examples

Basic Usage

Here is how to use this model to classify an image of the COCO 2017 dataset into one of the 1,000 ImageNet classes:

from transformers import BeitImageProcessor, BeitForImageClassification
from PIL import Image
import requests

url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
image = Image.open(requests.get(url, stream=True).raw)

processor = BeitImageProcessor.from_pretrained('microsoft/beit-base-patch16-224-pt22k-ft22k')
model = BeitForImageClassification.from_pretrained('microsoft/beit-base-patch16-224-pt22k-ft22k')

inputs = processor(images=image, return_tensors="pt")
outputs = model(**inputs)
logits = outputs.logits
# model predicts one of the 21,841 ImageNet-22k classes
predicted_class_idx = logits.argmax(-1).item()
print("Predicted class:", model.config.id2label[predicted_class_idx])

Currently, both the feature extractor and model support PyTorch.

🔧 Technical Details

Training data

The BEiT model was pre - trained on ImageNet - 21k, a dataset with 14 million images and 21k classes, and fine - tuned on the same dataset.

Training procedure

Preprocessing

The exact details of preprocessing of images during training/validation can be found here. Images are resized/rescaled to the same resolution (224x224) and normalized across the RGB channels with mean (0.5, 0.5, 0.5) and standard deviation (0.5, 0.5, 0.5).

Pretraining

For all pre - training related hyperparameters, refer to page 15 of the original paper.

Evaluation results

For evaluation results on several image classification benchmarks, refer to tables 1 and 2 of the original paper. Note that for fine - tuning, better results are obtained with a higher resolution. Increasing the model size will also lead to better performance.

BibTeX entry and citation info

@article{DBLP:journals/corr/abs-2106-08254,
  author    = {Hangbo Bao and
               Li Dong and
               Furu Wei},
  title     = {BEiT: {BERT} Pre-Training of Image Transformers},
  journal   = {CoRR},
  volume    = {abs/2106.08254},
  year      = {2021},
  url       = {https://arxiv.org/abs/2106.08254},
  archivePrefix = {arXiv},
  eprint    = {2106.08254},
  timestamp = {Tue, 29 Jun 2021 16:55:04 +0200},
  biburl    = {https://dblp.org/rec/journals/corr/abs-2106-08254.bib},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}

@inproceedings{deng2009imagenet,
  title={Imagenet: A large-scale hierarchical image database},
  author={Deng, Jia and Dong, Wei and Socher, Richard and Li, Li-Jia and Li, Kai and Fei-Fei, Li},
  booktitle={2009 IEEE conference on computer vision and pattern recognition},
  pages={248--255},
  year={2009},
  organization={Ieee}
}

📄 License

This model is released under the Apache - 2.0 license.