Webssl Dino3b Heavy2b 224

🚀 Web-SSL DINO ViT-3B: Heavy Filtered 2B MetaCLIP data, 224 Resolution

A 3 billion parameter Vision Transformer (ViT) trained with DINOv2 self - supervised learning on web - scale image data, offering strong performance in various vision tasks without language supervision.

🚀 Quick Start

To use the Web - SSL DINO 3B model, you can follow the code example below:

from transformers import AutoImageProcessor, Dinov2Model
import torch
from PIL import Image

processor = AutoImageProcessor.from_pretrained('facebook/webssl-dino3b-heavy2b-224')
model = Dinov2Model.from_pretrained('facebook/webssl-dino3b-heavy2b-224')

# Process an image
image = Image.open('path/to/image.jpg')
inputs = processor(images=image, return_tensors="pt")
with torch.no_grad():
    outputs = model(**inputs)

cls_features = outputs.last_hidden_state[:, 0]  # CLS token features
patch_features = outputs.last_hidden_state[:, 1:] # patch-wise token features

✨ Features

• Self - supervised Learning: Trained using self - supervised learning on heavily filtered web images without language supervision.
• Focused Filtering: The "heavy2b" filtering focuses on images with charts, tables, and documents, enhancing OCR & Chart understanding capabilities.
• Strong Performance: Matches or exceeds the performance of language - supervised models like CLIP across various vision tasks.

📦 Installation

The README does not provide specific installation steps, so this section is skipped.

💻 Usage Examples

Basic Usage

from transformers import AutoImageProcessor, Dinov2Model
import torch
from PIL import Image

processor = AutoImageProcessor.from_pretrained('facebook/webssl-dino3b-heavy2b-224')
model = Dinov2Model.from_pretrained('facebook/webssl-dino3b-heavy2b-224')

# Process an image
image = Image.open('path/to/image.jpg')
inputs = processor(images=image, return_tensors="pt")
with torch.no_grad():
    outputs = model(**inputs)

cls_features = outputs.last_hidden_state[:, 0]  # CLS token features
patch_features = outputs.last_hidden_state[:, 1:] # patch-wise token features

Advanced Usage

The README does not provide advanced usage examples, so this part is not shown.

📚 Documentation

Model Details

Model Descriptions

Web - SSL DINO 3B is a 3 billion parameter Vision Transformer model trained using self - supervised learning on heavily filtered web images without language supervision. The "heavy2b" designation indicates training on a subset of images containing charts, tables, and documents with readable text, representing only 1.3% of the original MetaCLIP dataset. This focused filtering significantly improves OCR & Chart understanding capabilities while maintaining strong performance on other vision tasks. This model demonstrates that pure visual learning, when scaled appropriately, can match or exceed the performance of language - supervised models like CLIP across various vision tasks.

🔧 Technical Details

The model is introduced in "Scaling Language - Free Visual Representation Learning" (Fan et al., 2025). It is a Vision Transformer (ViT) with 3 billion parameters, trained using DINOv2 self - supervised learning on heavily filtered web - scale image data without language supervision.

📄 License

The model is licensed under cc - by - nc - 4.0.

📄 Citation

@article{fan2025scaling,
  title={Scaling Language-Free Visual Representation Learning}, 
  author={David Fan and Shengbang Tong and Jiachen Zhu and Koustuv Sinha and Zhuang Liu and Xinlei Chen and Michael Rabbat and Nicolas Ballas and Yann LeCun and Amir Bar and Saining Xie},
  year={2025},
  eprint={2504.01017},
  archivePrefix={arXiv},
  primaryClass={cs.CV}
}