Rope Vit Reg4 B14 Capi Imagenet21k

🚀 rope_vit_reg4_b14_capi-imagenet21k模型卡片

這是一個基於RoPE的ViT圖像分類模型。該模型採用兩階段訓練流程：首先進行CAPI預訓練，然後在ImageNet-21K數據集上進行微調。

🚀 快速開始

本模型是一個RoPE ViT圖像分類模型，採用兩階段訓練，先進行CAPI預訓練，再在ImageNet-21K數據集上微調。

✨ 主要特性

• RoPE配置：本模型實現了EVA風格的旋轉位置嵌入（RoPE）。當處理與訓練分辨率（224x224）不同的分辨率時，可以通過配置pt_grid_size參數來優化模型行為。
  • 在較高分辨率下進行推理或進行“淺層”微調時，建議顯式設置pt_grid_size=(16, 16)（預訓練期間的默認網格大小）。
  • 在較高分辨率下進行激進微調時，將pt_grid_size保留為None，以使模型適應新的分辨率。

📦 安裝指南

文檔未提供安裝步驟，故跳過此章節。

💻 使用示例

基礎用法

圖像分類

import birder
from birder.inference.classification import infer_image

(net, model_info) = birder.load_pretrained_model("rope_vit_reg4_b14_capi-imagenet21k", inference=True)

# Get the image size the model was trained on
size = birder.get_size_from_signature(model_info.signature)

# Create an inference transform
transform = birder.classification_transform(size, model_info.rgb_stats)

image = "path/to/image.jpeg"  # or a PIL image, must be loaded in RGB format
(out, _) = infer_image(net, image, transform)
# out is a NumPy array with shape of (1, 19167), representing class probabilities.

圖像嵌入

import birder
from birder.inference.classification import infer_image

(net, model_info) = birder.load_pretrained_model("rope_vit_reg4_b14_capi-imagenet21k", inference=True)

# Get the image size the model was trained on
size = birder.get_size_from_signature(model_info.signature)

# Create an inference transform
transform = birder.classification_transform(size, model_info.rgb_stats)

image = "path/to/image.jpeg"  # or a PIL image
(out, embedding) = infer_image(net, image, transform, return_embedding=True)
# embedding is a NumPy array with shape of (1, 768)

檢測特徵圖

from PIL import Image
import birder

(net, model_info) = birder.load_pretrained_model("rope_vit_reg4_b14_capi-imagenet21k", inference=True)

# Get the image size the model was trained on
size = birder.get_size_from_signature(model_info.signature)

# Create an inference transform
transform = birder.classification_transform(size, model_info.rgb_stats)

image = Image.open("path/to/image.jpeg")
features = net.detection_features(transform(image).unsqueeze(0))
# features is a dict (stage name -> torch.Tensor)
print([(k, v.size()) for k, v in features.items()])
# Output example:
# [('neck', torch.Size([1, 768, 16, 16]))]

高級用法

設置pt_grid_size進行推理

# When running inference with a custom resolution (e.g., 336x336)
python predict.py --network rope_vit_reg4_b14 -t capi-imagenet21k --model-config '{"pt_grid_size":[16, 16]}' --size 336 ...

轉換具有顯式RoPE配置的模型

python tool.py convert-model --network rope_vit_reg4_b14 -t capi-imagenet21k --add-config '{"pt_grid_size":[16, 16]}'

📚 詳細文檔

模型詳情

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://arxiv.org/abs/2010.11929)
- [Rotary Position Embedding for Vision Transformer](https://arxiv.org/abs/2403.13298)
- [Vision Transformers Need Registers](https://arxiv.org/abs/2309.16588)
- [Cluster and Predict Latent Patches for Improved Masked Image Modeling](https://arxiv.org/abs/2502.08769) |

🔧 技術細節

文檔未提供足夠的技術實現細節（未超過50字），故跳過此章節。

📄 許可證

本項目採用Apache-2.0許可證。

📖 引用

@misc{dosovitskiy2021imageworth16x16words,
      title={An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale}, 
      author={Alexey Dosovitskiy and Lucas Beyer and Alexander Kolesnikov and Dirk Weissenborn and Xiaohua Zhai and Thomas Unterthiner and Mostafa Dehghani and Matthias Minderer and Georg Heigold and Sylvain Gelly and Jakob Uszkoreit and Neil Houlsby},
      year={2021},
      eprint={2010.11929},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2010.11929}, 
}

@misc{heo2024rotarypositionembeddingvision,
      title={Rotary Position Embedding for Vision Transformer},
      author={Byeongho Heo and Song Park and Dongyoon Han and Sangdoo Yun},
      year={2024},
      eprint={2403.13298},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2403.13298},
}

@misc{darcet2024visiontransformersneedregisters,
      title={Vision Transformers Need Registers}, 
      author={Timothée Darcet and Maxime Oquab and Julien Mairal and Piotr Bojanowski},
      year={2024},
      eprint={2309.16588},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2309.16588}, 
}

@misc{darcet2025clusterpredictlatentpatches,
      title={Cluster and Predict Latent Patches for Improved Masked Image Modeling},
      author={Timothée Darcet and Federico Baldassarre and Maxime Oquab and Julien Mairal and Piotr Bojanowski},
      year={2025},
      eprint={2502.08769},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2502.08769},
}