Tinyclip ViT 39M 16 Text 19M YFCC15M

🚀 TinyCLIP：通過親和度模仿和權重繼承實現CLIP蒸餾

TinyCLIP是一種用於大規模語言 - 圖像預訓練模型的新型跨模態蒸餾方法。該方法引入了兩項核心技術：親和度模仿和權重繼承。這項工作釋放了小型CLIP模型的潛力，充分利用了大規模模型以及預訓練數據，在速度和準確性之間取得了最佳平衡。

🚀 快速開始

與Transformers庫配合使用

from PIL import Image
import requests

from transformers import CLIPProcessor, CLIPModel

model = CLIPModel.from_pretrained("wkcn/TinyCLIP-ViT-39M-16-Text-19M-YFCC15M")
processor = CLIPProcessor.from_pretrained("wkcn/TinyCLIP-ViT-39M-16-Text-19M-YFCC15M")

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

inputs = processor(text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True)

outputs = model(**inputs)
logits_per_image = outputs.logits_per_image # this is the image-text similarity score
probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities

✨ 主要特性

特性亮點

• TinyCLIP ViT - 45M/32僅使用ViT - B/32一半的參數，就能實現相當的零樣本性能。
• TinyCLIP ResNet - 19M在參數減少50%的同時，推理速度提升了2倍，並在ImageNet上獲得了56.4%的準確率。

模型庫

注：具有自動繼承的模型配置是自動生成的。

💻 使用示例

基礎用法

from PIL import Image
import requests

from transformers import CLIPProcessor, CLIPModel

model = CLIPModel.from_pretrained("wkcn/TinyCLIP-ViT-39M-16-Text-19M-YFCC15M")
processor = CLIPProcessor.from_pretrained("wkcn/TinyCLIP-ViT-39M-16-Text-19M-YFCC15M")

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

inputs = processor(text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True)

outputs = model(**inputs)
logits_per_image = outputs.logits_per_image # this is the image-text similarity score
probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities

📚 詳細文檔

官方PyTorch實現

https://github.com/microsoft/Cream/tree/main/TinyCLIP

引用

如果這個倉庫對你有幫助，請考慮引用它。非常感謝！

@InProceedings{tinyclip,
    title     = {TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritance},
    author    = {Wu, Kan and Peng, Houwen and Zhou, Zhenghong and Xiao, Bin and Liu, Mengchen and Yuan, Lu and Xuan, Hong and Valenzuela, Michael and Chen, Xi (Stephen) and Wang, Xinggang and Chao, Hongyang and Hu, Han},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    month     = {October},
    year      = {2023},
    pages     = {21970-21980}
}

致謝

我們的代碼基於CLIP、OpenCLIP、CoFi和PyTorch。感謝貢獻者們的傑出貢獻！

📄 許可證

• 許可證