Aimv2 Large Patch14 448

🚀 图像特征提取模型transformers

本项目引入了基于多模态自回归目标进行预训练的AIMv2系列视觉模型。该模型训练和扩展简单直接，在多个多模态理解基准测试中表现出色，具有强大的识别性能。

🚀 快速开始

项目信息

模型评估结果

模型 aimv2-large-patch14-448 在多个数据集上的分类任务中表现如下：

✨ 主要特性

• 在大多数多模态理解基准测试中，性能优于OAI CLIP和SigLIP。
• 在开放词汇目标检测和指代表达理解任务中，性能优于DINOv2。
• 具有强大的识别性能，AIMv2 - 3B在使用冻结主干的情况下，在ImageNet上达到了89.5%的准确率。

💻 使用示例

基础用法

PyTorch

import requests
from PIL import Image
from transformers import AutoImageProcessor, AutoModel

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

processor = AutoImageProcessor.from_pretrained(
    "apple/aimv2-large-patch14-448",
)
model = AutoModel.from_pretrained(
    "apple/aimv2-large-patch14-448",
    trust_remote_code=True,
)

inputs = processor(images=image, return_tensors="pt")
outputs = model(**inputs)

JAX

import requests
from PIL import Image
from transformers import AutoImageProcessor, FlaxAutoModel

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

processor = AutoImageProcessor.from_pretrained(
    "apple/aimv2-large-patch14-448",
)
model = FlaxAutoModel.from_pretrained(
    "apple/aimv2-large-patch14-448",
    trust_remote_code=True,
)

inputs = processor(images=image, return_tensors="jax")
outputs = model(**inputs)

📄 许可证

本项目使用 apple-amlr 许可证。

📚 详细文档

• 论文链接：[AIMv2 Paper]
• 引用信息：

@misc{fini2024multimodalautoregressivepretraininglarge,
  author      = {Fini, Enrico and Shukor, Mustafa and Li, Xiujun and Dufter, Philipp and Klein, Michal and Haldimann, David and Aitharaju, Sai and da Costa, Victor Guilherme Turrisi and Béthune, Louis and Gan, Zhe and Toshev, Alexander T and Eichner, Marcin and Nabi, Moin and Yang, Yinfei and Susskind, Joshua M. and El-Nouby, Alaaeldin},
  url         = {https://arxiv.org/abs/2411.14402},
  eprint      = {2411.14402},
  eprintclass = {cs.CV},
  eprinttype  = {arXiv},
  title       = {Multimodal Autoregressive Pre-training of Large Vision Encoders},
  year        = {2024},
}