Webssl Dino2b Heavy2b 224

🚀 Web-SSL DINO ViT-2B: Heavy Filtered 2B MetaCLIP data, 224 Resolution

This is a 2-billion-parameter Vision Transformer (ViT) model. It is trained with DINOv2 self-supervised learning on heavily filtered web-scale image data, without any language supervision. This model is introduced in the paper "Scaling Language-Free Visual Representation Learning" (Fan et al., 2025).

✨ Features

• Trained using self - supervised learning on heavily filtered web images without language supervision.
• The "heavy2b" subset focuses on images with charts, tables, and documents with readable text, improving OCR & Chart understanding capabilities.
• Demonstrates that pure visual learning can match or exceed the performance of language - supervised models like CLIP on various vision tasks.

📦 Installation

The installation process is not provided in the original README. However, if you want to use this model, you need to have the transformers library installed. You can install it using the following command:

pip install transformers

💻 Usage Examples

Basic Usage

from transformers import AutoImageProcessor, Dinov2Model
import torch
from PIL import Image

processor = AutoImageProcessor.from_pretrained('facebook/webssl-dino2b-heavy2b-224')
model = Dinov2Model.from_pretrained('facebook/webssl-dino2b-heavy2b-224')

# Process an image
image = Image.open('path/to/image.jpg')
inputs = processor(images=image, return_tensors="pt")
with torch.no_grad():
    outputs = model(**inputs)

cls_features = outputs.last_hidden_state[:, 0]  # CLS token features
patch_features = outputs.last_hidden_state[:, 1:] # patch-wise token features

📚 Documentation

Model Details

Model Descriptions

Web - SSL DINO 2B is a 2 - billion - parameter Vision Transformer model. It is trained using self - supervised learning on heavily filtered web images without language supervision. The "heavy2b" designation means it is trained on a subset of images that explicitly contain charts, tables, and documents with readable text, which only accounts for 1.3% of the original MetaCLIP dataset. This focused filtering significantly enhances OCR & Chart understanding capabilities while maintaining strong performance on other vision tasks.

📄 License

This model is licensed under cc-by-nc-4.0.

📄 Citation

@article{fan2025scaling,
  title={Scaling Language-Free Visual Representation Learning}, 
  author={David Fan and Shengbang Tong and Jiachen Zhu and Koustuv Sinha and Zhuang Liu and Xinlei Chen and Michael Rabbat and Nicolas Ballas and Yann LeCun and Amir Bar and Saining Xie},
  year={2025},
  eprint={2504.01017},
  archivePrefix={arXiv},
  primaryClass={cs.CV}
}