Tinyclip ViT 61M 32 Text 29M LAION400M

Developed by wkcn

TinyCLIP is an innovative cross-modal distillation method for large-scale vision-language pretraining models, achieving optimal balance between speed and accuracy through affinity mimicking and weight inheritance techniques.

Text-to-Image

Transformers

Open Source License:MIT #Cross-modal distillation #Zero-shot classification #Lightweight CLIP

Downloads 950

Release Time : 12/19/2023

Model Overview

TinyCLIP is a cross-modal distillation approach that unlocks the potential of small-scale CLIP models through affinity mimicking and weight inheritance techniques, combining the advantages of large-scale models and pretraining data for efficient zero-shot image classification.

Model Features

Affinity mimicking

Preserves crucial semantic relationships in knowledge distillation by mimicking cross-modal affinity patterns from large CLIP models

Weight inheritance

Innovatively inherits weights from large models to significantly enhance small model performance

Efficient inference

Achieves 2x inference speedup with 50% fewer parameters while maintaining high performance

Model Capabilities

Zero-shot image classification

Cross-modal retrieval

Image-text matching

Use Cases

Image understanding

Image classification

Classifies images without fine-tuning

Achieves 62.4% accuracy on ImageNet

Cross-modal retrieval

Retrieves relevant images based on text descriptions or generates descriptions from images

Content moderation

Inappropriate content detection

Identifies potentially inappropriate content in images

license: mit datasets:

laion/laion400m pipeline_tag: zero-shot-image-classification tags:
tinyclip

TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritance

[ICCV 2023] - TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritance

TinyCLIP is a novel cross-modal distillation method for large-scale language-image pre-trained models. The method introduces two core techniques: affinity mimicking and weight inheritance. This work unleashes the capacity of small CLIP models, fully leveraging large-scale models as well as pre-training data and striking the best trade-off between speed and accuracy.

Use with Transformers

from PIL import Image
import requests

from transformers import CLIPProcessor, CLIPModel

model = CLIPModel.from_pretrained("wkcn/TinyCLIP-ViT-61M-32-Text-29M-LAION400M")
processor = CLIPProcessor.from_pretrained("wkcn/TinyCLIP-ViT-61M-32-Text-29M-LAION400M")

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

inputs = processor(text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True)

outputs = model(**inputs)
logits_per_image = outputs.logits_per_image # this is the image-text similarity score
probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities

Highlights

TinyCLIP ViT-45M/32 uses only half parameters of ViT-B/32 to achieves comparable zero-shot performance.
TinyCLIP ResNet-19M reduces the parameters by 50% while getting 2x inference speedup, and obtains 56.4% accuracy on ImageNet.

Model Zoo

Model	Weight inheritance	Pretrain	IN-1K Acc@1(%)	MACs(G)	Throughput(pairs/s)	Link
TinyCLIP ViT-39M/16 Text-19M	manual	YFCC-15M	63.5	9.5	1,469	Model
TinyCLIP ViT-8M/16 Text-3M	manual	YFCC-15M	41.1	2.0	4,150	Model
TinyCLIP ResNet-30M Text-29M	manual	LAION-400M	59.1	6.9	1,811	Model
TinyCLIP ResNet-19M Text-19M	manual	LAION-400M	56.4	4.4	3,024	Model
TinyCLIP ViT-61M/32 Text-29M	manual	LAION-400M	62.4	5.3	3,191	Model
TinyCLIP ViT-40M/32 Text-19M	manual	LAION-400M	59.8	3.5	4,641	Model
TinyCLIP ViT-63M/32 Text-31M	auto	LAION-400M	63.9	5.6	2,905	Model
TinyCLIP ViT-45M/32 Text-18M	auto	LAION-400M	61.4	3.7	3,682	Model
TinyCLIP ViT-22M/32 Text-10M	auto	LAION-400M	53.7	1.9	5,504	Model
TinyCLIP ViT-63M/32 Text-31M	auto	LAION+YFCC-400M	64.5	5.6	2,909	Model
TinyCLIP ViT-45M/32 Text-18M	auto	LAION+YFCC-400M	62.7	1.9	3,685	Model

Note: The configs of models with auto inheritance are generated automatically.

Official PyTorch Implementation

https://github.com/microsoft/Cream/tree/main/TinyCLIP

Citation

If this repo is helpful for you, please consider to cite it. :mega: Thank you! :)

@InProceedings{tinyclip,
    title     = {TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritance},
    author    = {Wu, Kan and Peng, Houwen and Zhou, Zhenghong and Xiao, Bin and Liu, Mengchen and Yuan, Lu and Xuan, Hong and Valenzuela, Michael and Chen, Xi (Stephen) and Wang, Xinggang and Chao, Hongyang and Hu, Han},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    month     = {October},
    year      = {2023},
    pages     = {21970-21980}
}

Acknowledge

Our code is based on CLIP, OpenCLIP, CoFi and PyTorch. Thank contributors for their awesome contribution!

License

License

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご