AIMV2-1B-patch14-336 Open-Source Vision Model - Strong Multimodal Understanding Ability and Excellent Test Performance

Aimv2 1B Patch14 336

Developed by apple

AIMv2 is a series of vision models pretrained with multimodal autoregressive objectives, achieving outstanding performance in multiple multimodal understanding benchmarks.

Image Classification #Multimodal Autoregressive Pretraining #High-Precision Image Classification #Open-Vocabulary Object Detection

Downloads 52

Release Time : 10/29/2024

Model Overview

AIMv2 is a vision model pretrained with multimodal autoregressive objectives, featuring robust image feature extraction and classification capabilities.

Model Features

Multimodal Autoregressive Pretraining

Pretrained with multimodal autoregressive objectives to enhance performance in multimodal understanding tasks.

High Performance

Outperforms OAI CLIP and SigLIP in multiple benchmarks, demonstrating strong recognition capabilities.

Broad Applicability

Delivers excellent performance across various datasets such as ImageNet, CIFAR, and Food101.

Model Capabilities

Image Feature Extraction

Image Classification

Multimodal Understanding

Use Cases

Computer Vision

Image Classification

Classifies images and is applicable to multiple datasets.

Achieves 88.7% accuracy on ImageNet-1k.

Object Detection

Excels in open-vocabulary object detection tasks.

Outperforms the DINOv2 model.

Medical Imaging

Pathological Image Analysis

Used for analyzing medical imaging data.

Achieves 94.2% accuracy on the Camelyon17 dataset.

🚀 AIMv2 Vision Models

AIMv2 is a family of vision models pre - trained with a multimodal autoregressive objective, offering high - performance in various vision tasks.

🚀 Quick Start

The AIMv2 family of vision models is pre - trained with a multimodal autoregressive objective. It's simple to train and scale effectively, and shows excellent performance in multiple benchmarks.

✨ Features

Multimodal Understanding: Outperforms OAI CLIP and SigLIP on the majority of multimodal understanding benchmarks.
Open - Vocabulary Tasks: Outperforms DINOv2 on open - vocabulary object detection and referring expression comprehension.
Strong Recognition: Exhibits strong recognition performance with AIMv2 - 3B achieving 89.5% on ImageNet using a frozen trunk.

💻 Usage Examples

Basic Usage

PyTorch

import requests
from PIL import Image
from transformers import AutoImageProcessor, AutoModel

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

processor = AutoImageProcessor.from_pretrained(
    "apple/aimv2-1B-patch14-336",
)
model = AutoModel.from_pretrained(
    "apple/aimv2-1B-patch14-336",
    trust_remote_code=True,
)

inputs = processor(images=image, return_tensors="pt")
outputs = model(**inputs)

JAX

import requests
from PIL import Image
from transformers import AutoImageProcessor, FlaxAutoModel

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

processor = AutoImageProcessor.from_pretrained(
    "apple/aimv2-1B-patch14-336",
)
model = FlaxAutoModel.from_pretrained(
    "apple/aimv2-1B-patch14-336",
    trust_remote_code=True,
)

inputs = processor(images=image, return_tensors="jax")
outputs = model(**inputs)

📚 Documentation

[AIMv2 Paper] [BibTeX]

We introduce the AIMv2 family of vision models pre - trained with a multimodal autoregressive objective. AIMv2 pre - training is simple and straightforward to train and scale effectively.

📄 License

The project uses the apple - amlr license.

📊 Performance Metrics

Property	Details
Library Name	transformers
Model Type	aimv2 - 1B - patch14 - 336
Pipeline Tag	image - feature - extraction
Tags	vision, image - feature - extraction, mlx, pytorch
Metrics	accuracy
Results
Task	Dataset
Classification	imagenet - 1k
Classification	inaturalist - 18
Classification	cifar10
Classification	cifar100
Classification	food101
Classification	dtd
Classification	oxford - pets
Classification	stanford - cars
Classification	camelyon17
Classification	patch - camelyon
Classification	rxrx1
Classification	eurosat
Classification	fmow
Classification	domainnet - infographic

📖 Citation

If you find our work useful, please consider citing us as:

@misc{fini2024multimodalautoregressivepretraininglarge,
  author      = {Fini, Enrico and Shukor, Mustafa and Li, Xiujun and Dufter, Philipp and Klein, Michal and Haldimann, David and Aitharaju, Sai and da Costa, Victor Guilherme Turrisi and Béthune, Louis and Gan, Zhe and Toshev, Alexander T and Eichner, Marcin and Nabi, Moin and Yang, Yinfei and Susskind, Joshua M. and El-Nouby, Alaaeldin},
  url         = {https://arxiv.org/abs/2411.14402},
  eprint      = {2411.14402},
  eprintclass = {cs.CV},
  eprinttype  = {arXiv},
  title       = {Multimodal Autoregressive Pre-training of Large Vision Encoders},
  year        = {2024},
}

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご