🚀 Marqo-FashionSigLIP模型卡片
Marqo-FashionSigLIP是一个多模态嵌入模型,与fashion clip相比,它在平均倒数排名(MRR)和召回率方面最多可提高57%。该模型利用了广义对比学习(GCL),这使得它不仅可以基于文本描述进行训练,还能结合类别、风格、颜色、材质、关键词和细节信息,从而为时尚产品搜索提供高度相关的结果。此模型是基于ViT - B - 16 - SigLIP(webli)进行微调得到的。
Github页面:Marqo-FashionCLIP
博客:Marqo博客

🚀 快速开始
本部分将介绍如何使用Marqo-FashionSigLIP模型,包括在Hugging Face、OpenCLIP和Transformers.js中的使用方法。
✨ 主要特性
- 性能提升:与fashion clip相比,在MRR和召回率方面最多可提高57%。
- 多模态训练:利用广义对比学习(GCL),可基于多种信息进行训练,提供更相关的搜索结果。
- 微调模型:基于ViT - B - 16 - SigLIP(webli)微调得到。
💻 使用示例
基础用法
Hugging Face
可以使用AutoModel
加载该模型:
from transformers import AutoModel, AutoProcessor
model = AutoModel.from_pretrained('Marqo/marqo-fashionSigLIP', trust_remote_code=True)
processor = AutoProcessor.from_pretrained('Marqo/marqo-fashionSigLIP', trust_remote_code=True)
import torch
from PIL import Image
image = [Image.open("docs/fashion-hippo.png")]
text = ["a hat", "a t-shirt", "shoes"]
processed = processor(text=text, images=image, padding='max_length', return_tensors="pt")
with torch.no_grad():
image_features = model.get_image_features(processed['pixel_values'], normalize=True)
text_features = model.get_text_features(processed['input_ids'], normalize=True)
text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
print("Label probs:", text_probs)
OpenCLIP
该模型可以与OpenCLIP无缝结合使用:
import open_clip
model, preprocess_train, preprocess_val = open_clip.create_model_and_transforms('hf-hub:Marqo/marqo-fashionSigLIP')
tokenizer = open_clip.get_tokenizer('hf-hub:Marqo/marqo-fashionSigLIP')
import torch
from PIL import Image
image = preprocess_val(Image.open("docs/fashion-hippo.png")).unsqueeze(0)
text = tokenizer(["a hat", "a t-shirt", "shoes"])
with torch.no_grad(), torch.cuda.amp.autocast():
image_features = model.encode_image(image, normalize=True)
text_features = model.encode_text(text, normalize=True)
text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
print("Label probs:", text_probs)
Transformers.js
你还可以使用Transformers.js库在JavaScript中运行该模型。首先,从NPM安装:
npm i @huggingface/transformers
然后,按如下方式计算嵌入:
import { SiglipTextModel, SiglipVisionModel, AutoTokenizer, AutoProcessor, RawImage, softmax, dot } from '@huggingface/transformers';
const model_id = 'Marqo/marqo-fashionSigLIP';
const tokenizer = await AutoTokenizer.from_pretrained(model_id);
const text_model = await SiglipTextModel.from_pretrained(model_id);
const processor = await AutoProcessor.from_pretrained(model_id);
const vision_model = await SiglipVisionModel.from_pretrained(model_id);
const texts = ['a hat', 'a t-shirt', 'shoes'];
const text_inputs = tokenizer(texts, { padding: 'max_length', truncation: true });
const { text_embeds } = await text_model(text_inputs);
const image = await RawImage.read('https://raw.githubusercontent.com/marqo-ai/marqo-FashionCLIP/main/docs/fashion-hippo.png');
const image_inputs = await processor(image);
const { image_embeds } = await vision_model(image_inputs);
const normalized_text_embeds = text_embeds.normalize().tolist();
const normalized_image_embeds = image_embeds.normalize().tolist()[0];
const text_probs = softmax(normalized_text_embeds.map((text_embed) =>
100.0 * dot(normalized_image_embeds, text_embed)
));
console.log(text_probs);
📚 详细文档
基准测试结果
以下报告了该模型在6个公开多模态时尚数据集(Atlas、DeepFashion (In - shop)、DeepFashion (Multimodal)、Fashion200k、KAGL和Polyvore)上的平均评估结果:
文本到图像(6个数据集的平均值)
模型 |
平均召回率 |
召回率@1 |
召回率@10 |
平均倒数排名(MRR) |
Marqo-FashionSigLIP |
0.231 |
0.121 |
0.340 |
0.239 |
FashionCLIP2.0 |
0.163 |
0.077 |
0.249 |
0.165 |
OpenFashionCLIP |
0.132 |
0.060 |
0.204 |
0.135 |
ViT-B-16-laion2b_s34b_b88k |
0.174 |
0.088 |
0.261 |
0.180 |
ViT-B-16-SigLIP-webli |
0.212 |
0.111 |
0.314 |
0.214 |
类别到产品(5个数据集的平均值)
模型 |
平均准确率 |
准确率@1 |
准确率@10 |
平均倒数排名(MRR) |
Marqo-FashionSigLIP |
0.737 |
0.758 |
0.716 |
0.812 |
FashionCLIP2.0 |
0.684 |
0.681 |
0.686 |
0.741 |
OpenFashionCLIP |
0.646 |
0.653 |
0.639 |
0.720 |
ViT-B-16-laion2b_s34b_b88k |
0.662 |
0.673 |
0.652 |
0.743 |
ViT-B-16-SigLIP-webli |
0.688 |
0.690 |
0.685 |
0.751 |
子类别到产品(4个数据集的平均值)
模型 |
平均准确率 |
准确率@1 |
准确率@10 |
平均倒数排名(MRR) |
Marqo-FashionSigLIP |
0.725 |
0.767 |
0.683 |
0.811 |
FashionCLIP2.0 |
0.657 |
0.676 |
0.638 |
0.733 |
OpenFashionCLIP |
0.598 |
0.619 |
0.578 |
0.689 |
ViT-B-16-laion2b_s34b_b88k |
0.638 |
0.651 |
0.624 |
0.712 |
ViT-B-16-SigLIP-webli |
0.643 |
0.643 |
0.643 |
0.726 |
📄 许可证
本项目采用Apache 2.0许可证。