Vit B 16 SigLIP2 256

🚀 ViT-B-16-SigLIP2-256模型卡片

本模型是一个基于WebLI数据集训练的SigLIP 2视觉语言模型，可用于零样本图像分类任务。它从原始的JAX检查点转换而来，适用于OpenCLIP库。

🚀 快速开始

本模型是在WebLI数据集上训练的SigLIP 2视觉语言模型。它已从Big Vision中的原始JAX检查点转换为可在OpenCLIP中使用的模型。

✨ 主要特性

• 模型类型：对比图像文本、零样本图像分类。
• 原始仓库：https://github.com/google-research/big_vision
• 数据集：WebLI
• 相关论文：
  • SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features: https://arxiv.org/abs/2502.14786
  • Sigmoid loss for language image pre-training: https://arxiv.org/abs/2303.15343

📦 安装指南

文档未提及安装步骤，跳过此章节。

💻 使用示例

基础用法

import torch
import torch.nn.functional as F
from urllib.request import urlopen
from PIL import Image
from open_clip import create_model_from_pretrained, get_tokenizer # works on open-clip-torch >= 2.31.0, timm >= 1.0.15

model, preprocess = create_model_from_pretrained('hf-hub:timm/ViT-B-16-SigLIP2-256')
tokenizer = get_tokenizer('hf-hub:timm/ViT-B-16-SigLIP2-256')

image = Image.open(urlopen(
    'https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/beignets-task-guide.png'
))
image = preprocess(image).unsqueeze(0)

labels_list = ["a dog", "a cat", "a donut", "a beignet"]
text = tokenizer(labels_list, context_length=model.context_length)

with torch.no_grad(), torch.cuda.amp.autocast():
    image_features = model.encode_image(image, normalize=True)
    text_features = model.encode_text(text, normalize=True)
    text_probs = torch.sigmoid(image_features @ text_features.T * model.logit_scale.exp() + model.logit_bias)

zipped_list = list(zip(labels_list, [100 * round(p.item(), 3) for p in text_probs[0]]))
print("Label probabilities: ", zipped_list)

📚 详细文档

文档未提供详细说明，跳过此章节。

🔧 技术细节

文档未提供技术实现细节，跳过此章节。

📄 许可证

本模型使用的许可证为apache-2.0。

📚 引用

@article{tschannen2025siglip,
  title={SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features},
  author={Tschannen, Michael and Gritsenko, Alexey and Wang, Xiao and Naeem, Muhammad Ferjad and Alabdulmohsin, Ibrahim and Parthasarathy, Nikhil and Evans, Talfan and Beyer, Lucas and Xia, Ye and Mustafa, Basil and H'enaff, Olivier and Harmsen, Jeremiah and Steiner, Andreas and Zhai, Xiaohua},
  year={2025},
  journal={arXiv preprint arXiv:2502.14786}
}        

@article{zhai2023sigmoid,
  title={Sigmoid loss for language image pre-training},
  author={Zhai, Xiaohua and Mustafa, Basil and Kolesnikov, Alexander and Beyer, Lucas},
  journal={arXiv preprint arXiv:2303.15343},
  year={2023}
}

@misc{big_vision,
  author = {Beyer, Lucas and Zhai, Xiaohua and Kolesnikov, Alexander},
  title = {Big Vision},
  year = {2022},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/google-research/big_vision}}
}