C RADIO

🚀 C-RADIO: Visual Feature Extraction Model

This model specializes in visual feature extraction. For example, RADIO can generate image embeddings that downstream models can utilize for image classification, offering significant value in the field of computer vision.

🚀 Quick Start

Installation

First, ensure you have the necessary libraries installed. You can install them using the following command:

pip install torch transformers pillow einops

Usage Example

import torch
from PIL import Image
from transformers import AutoModel, CLIPImageProcessor

hf_repo = "nvidia/C-RADIO"

image_processor = CLIPImageProcessor.from_pretrained(hf_repo)
model = AutoModel.from_pretrained(hf_repo, trust_remote_code=True)
model.eval().cuda()

image = Image.open('./assets/radio.png').convert('RGB')
pixel_values = image_processor(images=image, return_tensors='pt', do_resize=True).pixel_values
pixel_values = pixel_values.cuda()

summary, features = model(pixel_values)

Advanced Usage

from einops import rearrange
spatial_features = rearrange(spatial_features, 'b (h w) d -> b d h w', h=x.shape[-2] // patch_size, w=x.shape[-1] // patch_size)

✨ Features

• Feature Extraction: RADIO can generate image embeddings for downstream models to perform image classification.
• Flexible Output: It returns a tuple containing two tensors, summary and spatial_features, suitable for different tasks.

📦 Installation

You can install the required libraries using pip:

pip install torch transformers pillow einops

💻 Usage Examples

Basic Usage

import torch
from PIL import Image
from transformers import AutoModel, CLIPImageProcessor

hf_repo = "nvidia/C-RADIO"

image_processor = CLIPImageProcessor.from_pretrained(hf_repo)
model = AutoModel.from_pretrained(hf_repo, trust_remote_code=True)
model.eval().cuda()

image = Image.open('./assets/radio.png').convert('RGB')
pixel_values = image_processor(images=image, return_tensors='pt', do_resize=True).pixel_values
pixel_values = pixel_values.cuda()

summary, features = model(pixel_values)

Advanced Usage

from einops import rearrange
spatial_features = rearrange(spatial_features, 'b (h w) d -> b d h w', h=x.shape[-2] // patch_size, w=x.shape[-1] // patch_size)

📚 Documentation

Model Overview

This model performs visual feature extraction. For instance, RADIO generates image embeddings that can be used by a downstream model to classify images.

Model Architecture

Input

Output

Usage

RADIO will return a tuple with two tensors. The summary is similar to the cls_token in ViT and is meant to represent the general concept of the entire image. It has shape (B,C) with B being the batch dimension, and C being some number of channels. The spatial_features represent more localized content which should be suitable for dense tasks such as semantic segmentation, or for integration into an LLM.

Software Integration

Training, Testing, and Evaluation Datasets

Training Dataset

Evaluation Dataset

Inference

Ethical Considerations (For NVIDIA Models Only)

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse. Users should evaluate the model for safety and quality for a specific use case and build additional guardrails as appropriate.

Please report security vulnerabilities or NVIDIA AI Concerns here.

🔧 Technical Details

This model is based on the Vision Transformer architecture, which is a powerful neural network architecture for processing images. It takes RGB images as input and outputs image embeddings in tensor format. The input images should have pixel values in the [0, 1] range and can have resolutions up to 2048x2028 in increments of 16 pixels. The output embeddings can be used by downstream models for various tasks such as image classification.

📄 License

This model is governed by the NVIDIA Open Model License Agreement.

References

• AM - RADIO: Agglomerative Vision Foundation Model - Reduce All Domains Into One
• PHI - S: Distribution Balancing for Label - Free Multi - Teacher Distillation
• RADIO Amplified: Improved Baselines for Agglomerative Vision Foundation Models