Webssl Dino2b Light2b 224

🚀 Web-SSL DINO ViT-2B: Light Filtered 2B MetaCLIP data, 224 Resolution

A 2 billion parameter Vision Transformer (ViT) trained with DINOv2 self - supervised learning on lightly filtered web - scale image data without language supervision. It offers a new approach to visual representation learning.

🚀 Quick Start

Web-SSL DINO 2B is a powerful Vision Transformer model. You can quickly start using it with the following steps.

from transformers import AutoImageProcessor, Dinov2Model
import torch
from PIL import Image

processor = AutoImageProcessor.from_pretrained('facebook/webssl-dino2b-light2b-224')
model = Dinov2Model.from_pretrained('facebook/webssl-dino2b-light2b-224')

# Process an image
image = Image.open('path/to/image.jpg')
inputs = processor(images=image, return_tensors="pt")
with torch.no_grad():
    outputs = model(**inputs)

cls_features = outputs.last_hidden_state[:, 0]  # CLS token features
patch_features = outputs.last_hidden_state[:, 1:] # patch-wise token features

✨ Features

• Self - Supervised Learning: Trained using self - supervised learning on lightly filtered web images without language supervision.
• Enhanced Understanding: The filtering process improves OCR & Chart understanding capabilities.
• Strong Performance: Demonstrates comparable or superior performance to language - supervised models like CLIP across various vision tasks.

📦 Installation

The model can be installed using the transformers library. Make sure you have transformers installed in your Python environment. You can install it via pip:

pip install transformers

💻 Usage Examples

Basic Usage

from transformers import AutoImageProcessor, Dinov2Model
import torch
from PIL import Image

processor = AutoImageProcessor.from_pretrained('facebook/webssl-dino2b-light2b-224')
model = Dinov2Model.from_pretrained('facebook/webssl-dino2b-light2b-224')

# Process an image
image = Image.open('path/to/image.jpg')
inputs = processor(images=image, return_tensors="pt")
with torch.no_grad():
    outputs = model(**inputs)

cls_features = outputs.last_hidden_state[:, 0]  # CLS token features
patch_features = outputs.last_hidden_state[:, 1:] # patch-wise token features

📚 Documentation

Model Details

Model Descriptions

Web-SSL DINO 2B is a 2 billion parameter Vision Transformer model trained using self - supervised learning on lightly filtered web images without language supervision. The "light2b" designation indicates training on a subset of images containing any textual content, retaining approximately 50.3% of the original MetaCLIP dataset. This filtering improves OCR & Chart understanding capabilities while maintaining strong performance across all vision tasks. This model demonstrates that pure visual learning, when scaled appropriately, can match or exceed the performance of language - supervised models like CLIP across various vision tasks.

📄 License

This model is released under the cc-by-nc-4.0 license.

📚 Citation

@article{fan2025scaling,
  title={Scaling Language-Free Visual Representation Learning}, 
  author={David Fan and Shengbang Tong and Jiachen Zhu and Koustuv Sinha and Zhuang Liu and Xinlei Chen and Michael Rabbat and Nicolas Ballas and Yann LeCun and Amir Bar and Saining Xie},
  year={2025},
  eprint={2504.01017},
  archivePrefix={arXiv},
  primaryClass={cs.CV}
}