Vit Base Patch16 224.mae

🚀 vit_base_patch16_224.mae 模型卡

这是一个基于视觉变换器（ViT）的图像特征模型，使用自监督掩码自编码器（MAE）方法在 ImageNet-1k 数据集上进行了预训练，可用于图像特征提取等任务。

🚀 快速开始

本模型是一个基于视觉变换器（ViT）的图像特征模型，使用自监督掩码自编码器（MAE）方法在 ImageNet-1k 数据集上进行了预训练。下面将介绍其使用方法。

✨ 主要特性

• 模型类型：可用于图像分类或作为特征提取的骨干网络。
• 模型统计信息：
  • 参数数量（M）：85.8
  • GMACs：17.6
  • 激活值（M）：23.9
  • 图像尺寸：224 x 224
• 相关论文：
  • Masked Autoencoders Are Scalable Vision Learners
  • An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• 预训练数据集：ImageNet-1k
• 原始代码库：https://github.com/facebookresearch/mae

💻 使用示例

基础用法

图像分类

from urllib.request import urlopen
from PIL import Image
import timm

img = Image.open(urlopen(
    'https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/beignets-task-guide.png'
))

model = timm.create_model('vit_base_patch16_224.mae', pretrained=True)
model = model.eval()

# get model specific transforms (normalization, resize)
data_config = timm.data.resolve_model_data_config(model)
transforms = timm.data.create_transform(**data_config, is_training=False)

output = model(transforms(img).unsqueeze(0))  # unsqueeze single image into batch of 1

top5_probabilities, top5_class_indices = torch.topk(output.softmax(dim=1) * 100, k=5)

图像嵌入

from urllib.request import urlopen
from PIL import Image
import timm

img = Image.open(urlopen(
    'https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/beignets-task-guide.png'
))

model = timm.create_model(
    'vit_base_patch16_224.mae',
    pretrained=True,
    num_classes=0,  # remove classifier nn.Linear
)
model = model.eval()

# get model specific transforms (normalization, resize)
data_config = timm.data.resolve_model_data_config(model)
transforms = timm.data.create_transform(**data_config, is_training=False)

output = model(transforms(img).unsqueeze(0))  # output is (batch_size, num_features) shaped tensor

# or equivalently (without needing to set num_classes=0)

output = model.forward_features(transforms(img).unsqueeze(0))
# output is unpooled, a (1, 197, 768) shaped tensor

output = model.forward_head(output, pre_logits=True)
# output is a (1, num_features) shaped tensor

📚 详细文档

可在 timm 模型结果 中查看该模型的数据集和运行时指标。

📄 许可证

本模型采用 CC BY-NC 4.0 许可证。

📖 引用

@Article{MaskedAutoencoders2021,
  author  = {Kaiming He and Xinlei Chen and Saining Xie and Yanghao Li and Piotr Doll{'a}r and Ross Girshick},
  journal = {arXiv:2111.06377},
  title   = {Masked Autoencoders Are Scalable Vision Learners},
  year    = {2021},
}

@article{dosovitskiy2020vit,
  title={An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale},
  author={Dosovitskiy, Alexey and Beyer, Lucas and Kolesnikov, Alexander and Weissenborn, Dirk and Zhai, Xiaohua and Unterthiner, Thomas and  Dehghani, Mostafa and Minderer, Matthias and Heigold, Georg and Gelly, Sylvain and Uszkoreit, Jakob and Houlsby, Neil},
  journal={ICLR},
  year={2021}
}

@misc{rw2019timm,
  author = {Ross Wightman},
  title = {PyTorch Image Models},
  year = {2019},
  publisher = {GitHub},
  journal = {GitHub repository},
  doi = {10.5281/zenodo.4414861},
  howpublished = {\url{https://github.com/huggingface/pytorch-image-models}}
}