TinyCLIP Open-Source Cross-Modal Model - Efficiently Train Small-Scale CLIP Models for Free

Tinyclip ViT 40M 32 Text 19M LAION400M

Developed by wkcn

TinyCLIP is an innovative cross-modal distillation method for large-scale language-image pre-trained models, achieving efficient training of small-scale CLIP models through affinity mimicking and weight inheritance techniques.

Text-to-Image

Transformers

Open Source License:MIT #Zero-shot Image Classification #Cross-modal Distillation #Efficient Inference

Downloads 4,675

Release Time : 12/19/2023

Model Overview

TinyCLIP distills knowledge from large-scale CLIP models via affinity mimicking and weight inheritance techniques, significantly reducing model size while maintaining high performance.

Model Features

Affinity Mimicking

Knowledge distillation by mimicking cross-modal affinity relationships from large-scale CLIP models

Weight Inheritance

Inheriting weights from pre-trained large models to accelerate training and enhance performance

Efficient Inference

Reduces parameters by 50% compared to the original CLIP model while maintaining comparable performance

Model Capabilities

Zero-shot Image Classification

Cross-modal Retrieval

Image-Text Matching

Use Cases

Image Understanding

Image Classification

Classify images without fine-tuning

Achieves 59.8% accuracy on ImageNet

Cross-modal Retrieval

Retrieve relevant images based on text queries

Content Moderation

Inappropriate Content Detection

Detect whether images contain specific types of content

🚀 TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritance

TinyCLIP is a novel cross-modal distillation method for large-scale language-image pre-trained models. It introduces two core techniques: affinity mimicking and weight inheritance, unleashing the capacity of small CLIP models, leveraging large-scale models and pre-training data, and achieving an optimal balance between speed and accuracy.

🚀 Quick Start

Use with Transformers

from PIL import Image
import requests

from transformers import CLIPProcessor, CLIPModel

model = CLIPModel.from_pretrained("wkcn/TinyCLIP-ViT-40M-32-Text-19M-LAION400M")
processor = CLIPProcessor.from_pretrained("wkcn/TinyCLIP-ViT-40M-32-Text-19M-LAION400M")

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

inputs = processor(text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True)

outputs = model(**inputs)
logits_per_image = outputs.logits_per_image # this is the image-text similarity score
probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities

✨ Features

TinyCLIP ViT-45M/32 uses only half parameters of ViT-B/32 to achieve comparable zero-shot performance.
TinyCLIP ResNet-19M reduces the parameters by 50% while getting 2x inference speedup, and obtains 56.4% accuracy on ImageNet.

📚 Documentation

Model Zoo

Property	Details
Model Type	TinyCLIP ViT-39M/16 Text-19M, TinyCLIP ViT-8M/16 Text-3M, TinyCLIP ResNet-30M Text-29M, TinyCLIP ResNet-19M Text-19M, TinyCLIP ViT-61M/32 Text-29M, TinyCLIP ViT-40M/32 Text-19M, TinyCLIP ViT-63M/32 Text-31M, TinyCLIP ViT-45M/32 Text-18M, TinyCLIP ViT-22M/32 Text-10M, TinyCLIP ViT-63M/32 Text-31M, TinyCLIP ViT-45M/32 Text-18M
Weight inheritance	manual, auto
Pretrain	YFCC-15M, LAION-400M, LAION+YFCC-400M
IN-1K Acc@1(%)	63.5, 41.1, 59.1, 56.4, 62.4, 59.8, 63.9, 61.4, 53.7, 64.5, 62.7
MACs(G)	9.5, 2.0, 6.9, 4.4, 5.3, 3.5, 5.6, 3.7, 1.9, 5.6, 1.9
Throughput(pairs/s)	1,469, 4,150, 1,811, 3,024, 3,191, 4,641, 2,905, 3,682, 5,504, 2,909, 3,685
Link	Model, Model, Model, Model, Model, Model, Model, Model, Model, Model, Model

Note: The configs of models with auto inheritance are generated automatically.

Official PyTorch Implementation

https://github.com/microsoft/Cream/tree/main/TinyCLIP

Citation

If this repo is helpful for you, please consider to cite it. :mega: Thank you! :)

@InProceedings{tinyclip,
    title     = {TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritance},
    author    = {Wu, Kan and Peng, Houwen and Zhou, Zhenghong and Xiao, Bin and Liu, Mengchen and Yuan, Lu and Xuan, Hong and Valenzuela, Michael and Chen, Xi (Stephen) and Wang, Xinggang and Chao, Hongyang and Hu, Han},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    month     = {October},
    year      = {2023},
    pages     = {21970-21980}
}

Acknowledge

Our code is based on CLIP, OpenCLIP, CoFi and PyTorch. Thank contributors for their awesome contribution!

📄 License

License

[ICCV 2023] - TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritance

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご