AIMV2-1B-Patch14-224 Open-Source Vision Model - Multimodal Pretraining Empowers Vision Task Processing

Aimv2 1B Patch14 224

Developed by apple

AIMv2 is a series of vision models pretrained with multimodal autoregressive objectives, excelling in various vision tasks.

Image Classification #Multimodal Autoregressive Pretraining #High-Precision Image Classification #Open-Vocabulary Detection

Downloads 299

Release Time : 10/29/2024

Model Overview

AIMv2 is a large-scale visual encoder based on multimodal autoregressive pretraining, primarily used for image feature extraction and classification tasks. The model demonstrates outstanding performance across multiple benchmarks, particularly surpassing peer models in open-vocabulary object detection and referring expression comprehension tasks.

Model Features

Multimodal Autoregressive Pretraining

Utilizes innovative multimodal autoregressive objectives for pretraining, effectively enhancing model performance.

Exceptional Classification Performance

Achieves over 88% accuracy on classification tasks such as ImageNet-1k.

Cross-Task Generalization Capability

Outperforms DINOv2 in open-vocabulary object detection and referring expression comprehension tasks.

Model Capabilities

Image feature extraction

Image classification

Multimodal understanding

Open-vocabulary object detection

Referring expression comprehension

Use Cases

Computer Vision

General Image Classification

Used for standard image classification tasks such as ImageNet.

88.1% accuracy on ImageNet-1k

Fine-Grained Classification

Used for fine-grained image classification, such as the Stanford Cars dataset.

96.5% accuracy on the Stanford Cars dataset

Medical Image Analysis

Used for medical image classification, such as the Camelyon17 dataset.

94.2% accuracy on Camelyon17

Remote Sensing Image Processing

Satellite Image Classification

Used for satellite image classification tasks such as EuroSAT.

98.8% accuracy on EuroSAT

🚀 AIMv2 Vision Models

The AIMv2 family of vision models is pre - trained with a multimodal autoregressive objective, offering high - performance solutions for various vision tasks.

🚀 Quick Start

The AIMv2 models are designed to be simple and straightforward to train and scale effectively. They offer excellent performance in multiple multimodal understanding benchmarks and vision tasks.

✨ Features

Outperforms OAI CLIP and SigLIP on the majority of multimodal understanding benchmarks.
Outperforms DINOv2 on open - vocabulary object detection and referring expression comprehension.
Exhibits strong recognition performance with AIMv2 - 3B achieving 89.5% on ImageNet using a frozen trunk.

📦 Installation

The document doesn't provide specific installation steps, so this section is skipped.

💻 Usage Examples

Basic Usage

PyTorch

import requests
from PIL import Image
from transformers import AutoImageProcessor, AutoModel

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

processor = AutoImageProcessor.from_pretrained(
    "apple/aimv2-1B-patch14-224",
)
model = AutoModel.from_pretrained(
    "apple/aimv2-1B-patch14-224",
    trust_remote_code=True,
)

inputs = processor(images=image, return_tensors="pt")
outputs = model(**inputs)

JAX

import requests
from PIL import Image
from transformers import AutoImageProcessor, FlaxAutoModel

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

processor = AutoImageProcessor.from_pretrained(
    "apple/aimv2-1B-patch14-224",
)
model = FlaxAutoModel.from_pretrained(
    "apple/aimv2-1B-patch14-224",
    trust_remote_code=True,
)

inputs = processor(images=image, return_tensors="jax")
outputs = model(**inputs)

📚 Documentation

[AIMv2 Paper] [BibTeX]

🔧 Technical Details

Property	Details
Library Name	transformers
Model Type	aimv2 - 1B - patch14 - 224
Metrics	accuracy
Pipeline Tag	image - feature - extraction
Tags	vision, image - feature - extraction, mlx, pytorch
License	apple - amlr

Results on Different Datasets

Task	Dataset	Accuracy	Verified
Classification	imagenet - 1k	88.1	false
Classification	inaturalist - 18	79.7	false
Classification	cifar10	99.4	false
Classification	cifar100	94.1	false
Classification	food101	96.7	false
Classification	dtd	88.4	false
Classification	oxford - pets	96.8	false
Classification	stanford - cars	96.5	false
Classification	camelyon17	94.2	false
Classification	patch - camelyon	89.0	false
Classification	rxrx1	6.7	false
Classification	eurosat	98.8	false
Classification	fmow	63.2	false
Classification	domainnet - infographic	71.7	false

📄 License

The project uses the apple - amlr license.

📖 Citation

If you find our work useful, please consider citing us as:

@misc{fini2024multimodalautoregressivepretraininglarge,
  author      = {Fini, Enrico and Shukor, Mustafa and Li, Xiujun and Dufter, Philipp and Klein, Michal and Haldimann, David and Aitharaju, Sai and da Costa, Victor Guilherme Turrisi and Béthune, Louis and Gan, Zhe and Toshev, Alexander T and Eichner, Marcin and Nabi, Moin and Yang, Yinfei and Susskind, Joshua M. and El - Nouby, Alaaeldin},
  url         = {https://arxiv.org/abs/2411.14402},
  eprint      = {2411.14402},
  eprintclass = {cs.CV},
  eprinttype  = {arXiv},
  title       = {Multimodal Autoregressive Pre - training of Large Vision Encoders},
  year        = {2024},
}

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご