Samvit Base Patch16.sa1b

🚀 samvit_base_patch16.sa1b模型卡片

这是一个Segment-Anything Vision Transformer (SAM ViT)图像特征模型（注意：用于提取特征和微调，不包含分割头）。该模型由论文作者在SA - 1B数据集上进行预训练以用于分割任务，并使用MAE的权重进行初始化。

🚀 快速开始

本模型是一个基于Transformer架构的图像特征提取模型，可用于图像分类和获取图像嵌入等任务。

✨ 主要特性

• 模型类型：图像分类/特征骨干网络
• 模型统计信息：
  • 参数数量（M）：89.7
  • GMACs：486.4
  • 激活值数量（M）：1343.3
  • 图像尺寸：1024 x 1024
• 相关论文：
  • Segment Anything
  • An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• 原始代码库：https://github.com/facebookresearch/segment - anything
• 预训练数据集：SA - 1B

📦 安装指南

文档未提及安装步骤，若需使用该模型，可参考timm库的官方安装说明。

💻 使用示例

基础用法

图像分类

from urllib.request import urlopen
from PIL import Image
import timm

img = Image.open(urlopen(
    'https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/beignets-task-guide.png'
))

model = timm.create_model('samvit_base_patch16.sa1b', pretrained=True)
model = model.eval()

# get model specific transforms (normalization, resize)
data_config = timm.data.resolve_model_data_config(model)
transforms = timm.data.create_transform(**data_config, is_training=False)

output = model(transforms(img).unsqueeze(0))  # unsqueeze single image into batch of 1

top5_probabilities, top5_class_indices = torch.topk(output.softmax(dim=1) * 100, k=5)

图像嵌入

from urllib.request import urlopen
from PIL import Image
import timm

img = Image.open(urlopen(
    'https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/beignets-task-guide.png'
))

model = timm.create_model(
    'samvit_base_patch16.sa1b',
    pretrained=True,
    num_classes=0,  # remove classifier nn.Linear
)
model = model.eval()

# get model specific transforms (normalization, resize)
data_config = timm.data.resolve_model_data_config(model)
transforms = timm.data.create_transform(**data_config, is_training=False)

output = model(transforms(img).unsqueeze(0))  # output is (batch_size, num_features) shaped tensor

# or equivalently (without needing to set num_classes=0)

output = model.forward_features(transforms(img).unsqueeze(0))
# output is unpooled, a (1, 256, 64, 64) shaped tensor

output = model.forward_head(output, pre_logits=True)
# output is a (1, num_features) shaped tensor

📚 详细文档

可在timm 模型结果中探索该模型的数据集和运行时指标。

📄 许可证

本项目采用Apache - 2.0许可证。

📚 引用

@article{kirillov2023segany,
  title={Segment Anything},
  author={Kirillov, Alexander and Mintun, Eric and Ravi, Nikhila and Mao, Hanzi and Rolland, Chloe and Gustafson, Laura and Xiao, Tete and Whitehead, Spencer and Berg, Alexander C. and Lo, Wan-Yen and Doll{'a}r, Piotr and Girshick, Ross},
  journal={arXiv:2304.02643},
  year={2023}
}

@article{dosovitskiy2020vit,
  title={An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale},
  author={Dosovitskiy, Alexey and Beyer, Lucas and Kolesnikov, Alexander and Weissenborn, Dirk and Zhai, Xiaohua and Unterthiner, Thomas and  Dehghani, Mostafa and Minderer, Matthias and Heigold, Georg and Gelly, Sylvain and Uszkoreit, Jakob and Houlsby, Neil},
  journal={ICLR},
  year={2021}
}

@misc{rw2019timm,
  author = {Ross Wightman},
  title = {PyTorch Image Models},
  year = {2019},
  publisher = {GitHub},
  journal = {GitHub repository},
  doi = {10.5281/zenodo.4414861},
  howpublished = {\url{https://github.com/huggingface/pytorch-image-models}}
}