Videomae Huge Finetuned Kinetics

🚀 VideoMAE (huge-sized model, fine-tuned on Kinetics-400)

VideoMAE is a pre - trained model for video processing. It is pre - trained in a self - supervised way and fine - tuned on Kinetics - 400 for video classification tasks.

🚀 Quick Start

The VideoMAE model was pre - trained for 1600 epochs in a self - supervised manner and then fine - tuned in a supervised way on Kinetics - 400. It was introduced in the paper VideoMAE: Masked Autoencoders are Data - Efficient Learners for Self - Supervised Video Pre - Training by Tong et al. and first released in this repository.

Disclaimer: The team releasing VideoMAE did not write a model card for this model, so this model card has been written by the Hugging Face team.

✨ Features

Model description

VideoMAE extends Masked Autoencoders (MAE) to video. Its architecture is quite similar to a standard Vision Transformer (ViT), with a decoder on top for predicting pixel values of masked patches.

Videos are presented to the model as a sequence of fixed - size patches (resolution 16x16), which are linearly embedded. A [CLS] token is added to the start of the sequence for classification tasks. Fixed sinus/cosinus position embeddings are also added before feeding the sequence into the Transformer encoder layers.

Through pre - training, the model learns an internal representation of videos, which can be used to extract features for downstream tasks. For example, if you have a labeled video dataset, you can train a standard classifier by adding a linear layer on top of the pre - trained encoder. Usually, a linear layer is added on top of the [CLS] token, as the last hidden state of this token can represent an entire video.

Intended uses & limitations

You can use the raw model for video classification into one of the 400 possible Kinetics - 400 labels.

📦 Installation

No specific installation steps are provided in the original document.

💻 Usage Examples

Basic Usage

Here is how to use this model to classify a video:

from transformers import VideoMAEImageProcessor, VideoMAEForVideoClassification
import numpy as np
import torch

video = list(np.random.randn(16, 3, 224, 224))

processor = VideoMAEImageProcessor.from_pretrained("MCG-NJU/videomae-huge-finetuned-kinetics")
model = VideoMAEForVideoClassification.from_pretrained("MCG-NJU/videomae-huge-finetuned-kinetics")

inputs = processor(video, return_tensors="pt")

with torch.no_grad():
  outputs = model(**inputs)
  logits = outputs.logits

predicted_class_idx = logits.argmax(-1).item()
print("Predicted class:", model.config.id2label[predicted_class_idx])

For more code examples, we refer to the documentation.

📚 Documentation

Training data

(to do, feel free to open a PR)

Training procedure

Preprocessing

(to do, feel free to open a PR)

Pretraining

(to do, feel free to open a PR)

Evaluation results

This model obtains a top - 1 accuracy of 86.6 and a top - 5 accuracy of 97.1 on the test set of Kinetics - 400.

BibTeX entry and citation info

misc{https://doi.org/10.48550/arxiv.2203.12602,
  doi = {10.48550/ARXIV.2203.12602},
  url = {https://arxiv.org/abs/2203.12602},
  author = {Tong, Zhan and Song, Yibing and Wang, Jue and Wang, Limin},
  keywords = {Computer Vision and Pattern Recognition (cs.CV), FOS: Computer and information sciences, FOS: Computer and information sciences},
  title = {VideoMAE: Masked Autoencoders are Data - Efficient Learners for Self - Supervised Video Pre - Training},
  publisher = {arXiv},
  year = {2022},
  copyright = {Creative Commons Attribution 4.0 International}
}

📄 License

The model is licensed under "cc - by - nc - 4.0".