Webssl Dino300m Full2b 224

🚀 Web-SSL DINO ViT-300M: 2B MetaCLIP data, 224 Resolution

A 300 million parameter Vision Transformer (ViT) trained with DINOv2 self-supervised learning on web-scale image data without language supervision.

🚀 Quick Start

Web-SSL DINO 300M is a 300 million parameter Vision Transformer model trained using self-supervised learning on 2 billion web images without language supervision. This model demonstrates that pure visual learning, when scaled appropriately, can match or exceed the performance of language-supervised models like CLIP across various vision tasks.

✨ Features

• Architecture: ViT (1536 width, 40 depth, 24 heads)
• Parameters: 300M
• Resolution: 224×224 pixels
• Training: Self-supervised Web-DINO on 2B image samples from MetaCLIP web data

Here is an overview of the WebSSL Model:

📦 Installation

No specific installation steps are provided in the original README.

💻 Usage Examples

Basic Usage

from transformers import AutoImageProcessor, Dinov2Model
import torch
from PIL import Image

processor = AutoImageProcessor.from_pretrained('facebook/webssl-dino300m-full2b-224')
model = Dinov2Model.from_pretrained('facebook/webssl-dino300m-full2b-224')

# Process an image
image = Image.open('path/to/image.jpg')
inputs = processor(images=image, return_tensors="pt")
with torch.no_grad():
    outputs = model(**inputs)

cls_features = outputs.last_hidden_state[:, 0]  # CLS token features
patch_features = outputs.last_hidden_state[:, 1:] # patch-wise token features

📚 Documentation

The model was introduced in "Scaling Language-Free Visual Representation Learning" (Fan et al., 2025).

📄 License

This project is licensed under the CC BY-NC 4.0 license.

📚 Citation

@article{fan2025scaling,
  title={Scaling Language-Free Visual Representation Learning}, 
  author={David Fan and Shengbang Tong and Jiachen Zhu and Koustuv Sinha and Zhuang Liu and Xinlei Chen and Michael Rabbat and Nicolas Ballas and Yann LeCun and Amir Bar and Saining Xie},
  year={2025},
  eprint={2504.01017},
  archivePrefix={arXiv},
  primaryClass={cs.CV}
}