Siglip2 So400m Patch14 224

🚀 SigLIP 2 So400m

SigLIP 2 擴展了 SigLIP 的預訓練目標，將先前獨立開發的技術整合為一個統一的方案，以提升語義理解、定位和密集特徵提取能力。

🚀 快速開始

你可以使用該原始模型進行零樣本圖像分類和圖像 - 文本檢索等任務，也可以將其作為視覺語言模型（VLM）的視覺編碼器（以及用於其他視覺任務）。

以下是使用此模型進行零樣本圖像分類的示例：

from transformers import pipeline

# load pipeline
ckpt = "google/siglip2-so400m-patch14-224"
image_classifier = pipeline(model=ckpt, task="zero-shot-image-classification")

# load image and candidate labels
url = "http://images.cocodataset.org/val2017/000000039769.jpg"
candidate_labels = ["2 cats", "a plane", "a remote"]

# run inference
outputs = image_classifier(image, candidate_labels)
print(outputs)

你還可以像這樣使用視覺塔對圖像進行編碼：

import torch
from transformers import AutoModel, AutoProcessor
from transformers.image_utils import load_image

# load the model and processor
ckpt = "google/siglip2-so400m-patch14-224"
model = AutoModel.from_pretrained(ckpt, device_map="auto").eval()
processor = AutoProcessor.from_pretrained(ckpt)

# load the image
image = load_image("https://huggingface.co/datasets/merve/coco/resolve/main/val2017/000000000285.jpg")
inputs = processor(images=[image], return_tensors="pt").to(model.device)

# run infernece
with torch.no_grad():
    image_embeddings = model.get_image_features(**inputs)    

print(image_embeddings.shape)

更多代碼示例，請參考 siglip 文檔。

💻 使用示例

基礎用法

from transformers import pipeline

# load pipeline
ckpt = "google/siglip2-so400m-patch14-224"
image_classifier = pipeline(model=ckpt, task="zero-shot-image-classification")

# load image and candidate labels
url = "http://images.cocodataset.org/val2017/000000039769.jpg"
candidate_labels = ["2 cats", "a plane", "a remote"]

# run inference
outputs = image_classifier(image, candidate_labels)
print(outputs)

高級用法

import torch
from transformers import AutoModel, AutoProcessor
from transformers.image_utils import load_image

# load the model and processor
ckpt = "google/siglip2-so400m-patch14-224"
model = AutoModel.from_pretrained(ckpt, device_map="auto").eval()
processor = AutoProcessor.from_pretrained(ckpt)

# load the image
image = load_image("https://huggingface.co/datasets/merve/coco/resolve/main/val2017/000000000285.jpg")
inputs = processor(images=[image], return_tensors="pt").to(model.device)

# run infernece
with torch.no_grad():
    image_embeddings = model.get_image_features(**inputs)    

print(image_embeddings.shape)

🔧 技術細節

訓練過程

SigLIP 2 在 SigLIP 的基礎上增加了一些巧妙的訓練目標：

1. 解碼器損失
2. 全局 - 局部和掩碼預測損失
3. 寬高比和分辨率適應性

訓練數據

SigLIP 2 在 WebLI 數據集 (Chen et al., 2023) 上進行預訓練。

計算資源

該模型在多達 2048 個 TPU - v5e 芯片上進行訓練。

📚 詳細文檔

評估結果

以下是 SigLIP 2 的評估結果（取自論文）。

BibTeX 引用和引用信息

@misc{tschannen2025siglip2multilingualvisionlanguage,
      title={SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features}, 
      author={Michael Tschannen and Alexey Gritsenko and Xiao Wang and Muhammad Ferjad Naeem and Ibrahim Alabdulmohsin and Nikhil Parthasarathy and Talfan Evans and Lucas Beyer and Ye Xia and Basil Mustafa and Olivier Hénaff and Jeremiah Harmsen and Andreas Steiner and Xiaohua Zhai},
      year={2025},
      eprint={2502.14786},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2502.14786}, 
}

📄 許可證

本項目採用 Apache-2.0 許可證。