Marqo Fashionsiglip

🚀 Marqo-FashionSigLIP模型卡片

Marqo-FashionSigLIP利用廣義對比學習（GCL），使模型不僅能基於文本描述進行訓練，還能結合類別、風格、顏色、材質、關鍵詞和細節信息，從而為時尚產品提供高度相關的搜索結果。該模型是在ViT - B - 16 - SigLIP (webli)的基礎上進行微調得到的。

GitHub頁面：Marqo-FashionCLIP

博客：Marqo博客

🚀 快速開始

✨ 主要特性

• 支持標籤：clip、e - commerce、fashion、multimodal retrieval、siglip
• 支持的庫：OpenCLIP
• 任務類型：零樣本圖像分類
• 許可證：Apache - 2.0
• 訓練數據集：Marqo/atlas、Marqo/deepfashion - inshop、Marqo/deepfashion - multimodal、Marqo/fashion200k、Marqo/iMaterialist、Marqo/KAGL、Marqo/polyvore
• 評估指標：precision、recall、MRR

📦 安裝指南

此模型可通過OpenCLIP無縫使用，示例代碼如下：

import open_clip
model, preprocess_train, preprocess_val = open_clip.create_model_and_transforms('hf-hub:Marqo/marqo-fashionSigLIP')
tokenizer = open_clip.get_tokenizer('hf-hub:Marqo/marqo-fashionSigLIP')

import torch
from PIL import Image

image = preprocess_val(Image.open("docs/fashion-hippo.png")).unsqueeze(0)
text = tokenizer(["a hat", "a t-shirt", "shoes"])

with torch.no_grad(), torch.cuda.amp.autocast():
    image_features = model.encode_image(image)
    text_features = model.encode_text(text)
    image_features /= image_features.norm(dim=-1, keepdim=True)
    text_features /= text_features.norm(dim=-1, keepdim=True)

    text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)

print("Label probs:", text_probs)

💻 使用示例

基礎用法

import open_clip
model, preprocess_train, preprocess_val = open_clip.create_model_and_transforms('hf-hub:Marqo/marqo-fashionSigLIP')
tokenizer = open_clip.get_tokenizer('hf-hub:Marqo/marqo-fashionSigLIP')

import torch
from PIL import Image

image = preprocess_val(Image.open("docs/fashion-hippo.png")).unsqueeze(0)
text = tokenizer(["a hat", "a t-shirt", "shoes"])

with torch.no_grad(), torch.cuda.amp.autocast():
    image_features = model.encode_image(image)
    text_features = model.encode_text(text)
    image_features /= image_features.norm(dim=-1, keepdim=True)
    text_features /= text_features.norm(dim=-1, keepdim=True)

    text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)

print("Label probs:", text_probs)

📚 詳細文檔

基準測試結果

以下是該模型在6個公開多模態時尚數據集（Atlas、[DeepFashion (In - shop)](https://huggingface.co/datasets/Marqo/deepfashion - inshop)、[DeepFashion (Multimodal)](https://huggingface.co/datasets/Marqo/deepfashion - multimodal)、Fashion200k、KAGL和Polyvore）上的平均評估結果：

文本到圖像（6個數據集的平均值）

類別到產品（5個數據集的平均值）

子類別到產品（4個數據集的平均值）

📄 許可證

本項目採用Apache - 2.0許可證。