Aimv2 Large Patch14 224

🚀 圖像特徵提取模型AIMv2

AIMv2是一系列經過多模態自迴歸目標預訓練的視覺模型，訓練簡單直接，可有效擴展。該模型在多數多模態理解基準測試中表現出色，在開放詞彙目標檢測和指代表達理解等任務中也有優秀表現。

🚀 快速開始

模型信息

模型效果

模型亮點

1. 在多數多模態理解基準測試中，性能優於OAI CLIP和SigLIP。
2. 在開放詞彙目標檢測和指代表達理解任務上，表現優於DINOv2。
3. AIMv2 - 3B模型使用凍結主幹在ImageNet上達到89.5%的準確率。

模型概覽圖

💻 使用示例

基礎用法 - PyTorch

import requests
from PIL import Image
from transformers import AutoImageProcessor, AutoModel

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

processor = AutoImageProcessor.from_pretrained(
    "apple/aimv2-large-patch14-224",
)
model = AutoModel.from_pretrained(
    "apple/aimv2-large-patch14-224",
    trust_remote_code=True,
)

inputs = processor(images=image, return_tensors="pt")
outputs = model(**inputs)

高級用法 - JAX

import requests
from PIL import Image
from transformers import AutoImageProcessor, FlaxAutoModel

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

processor = AutoImageProcessor.from_pretrained(
    "apple/aimv2-large-patch14-224",
)
model = FlaxAutoModel.from_pretrained(
    "apple/aimv2-large-patch14-224",
    trust_remote_code=True,
)

inputs = processor(images=image, return_tensors="jax")
outputs = model(**inputs)

📚 詳細文檔

• 論文鏈接：[AIMv2 Paper]
• 引用信息：

@misc{fini2024multimodalautoregressivepretraininglarge,
  author      = {Fini, Enrico and Shukor, Mustafa and Li, Xiujun and Dufter, Philipp and Klein, Michal and Haldimann, David and Aitharaju, Sai and da Costa, Victor Guilherme Turrisi and Béthune, Louis and Gan, Zhe and Toshev, Alexander T and Eichner, Marcin and Nabi, Moin and Yang, Yinfei and Susskind, Joshua M. and El-Nouby, Alaaeldin},
  url         = {https://arxiv.org/abs/2411.14402},
  eprint      = {2411.14402},
  eprintclass = {cs.CV},
  eprinttype  = {arXiv},
  title       = {Multimodal Autoregressive Pre-training of Large Vision Encoders},
  year        = {2024},
}

如果你覺得我們的工作有幫助，請考慮引用我們的論文。