🚀 clip-vit-base-patch32-ko
A Korean CLIP model trained using the method described in Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
Widget
- src: https://huggingface.co/datasets/mishig/sample_images/resolve/main/cat-dog-music.png
- candidate_labels: 기타치는 고양이, 피아노 치는 강아지
- example_title: Guitar, cat and dog
License
This project is licensed under the MIT license.
🚀 Quick Start
This is a Korean CLIP model trained by Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation.
Training code: https://github.com/Bing-su/KoCLIP_training_code
Data used: All Korean-English parallel data from AIHUB
💻 Usage Examples
Basic Usage
import requests
import torch
from PIL import Image
from transformers import AutoModel, AutoProcessor
repo = "Bingsu/clip-vit-base-patch32-ko"
model = AutoModel.from_pretrained(repo)
processor = AutoProcessor.from_pretrained(repo)
url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)
inputs = processor(text=["고양이 두 마리", "개 두 마리"], images=image, return_tensors="pt", padding=True)
with torch.inference_mode():
outputs = model(**inputs)
logits_per_image = outputs.logits_per_image
probs = logits_per_image.softmax(dim=1)
>>> probs
tensor([[0.9926, 0.0074]])
Advanced Usage
from transformers import pipeline
repo = "Bingsu/clip-vit-base-patch32-ko"
pipe = pipeline("zero-shot-image-classification", model=repo)
url = "http://images.cocodataset.org/val2017/000000039769.jpg"
result = pipe(images=url, candidate_labels=["고양이 한 마리", "고양이 두 마리", "분홍색 소파에 드러누운 고양이 친구들"], hypothesis_template="{}")
>>> result
[{'score': 0.9456236958503723, 'label': '분홍색 소파에 드러누운 고양이 친구들'},
{'score': 0.05315302312374115, 'label': '고양이 두 마리'},
{'score': 0.0012233294546604156, 'label': '고양이 한 마리'}]
🔧 Technical Details
Tokenizer
The tokenizer was trained by mixing Korean and English data at a ratio of 7:3 and using .train_new_from_iterator
from the original CLIP tokenizer.
https://github.com/huggingface/transformers/blob/bc21aaca789f1a366c05e8b5e111632944886393/src/transformers/models/clip/modeling_clip.py#L661-L666
pooled_output = last_hidden_state[
torch.arange(last_hidden_state.shape[0]), input_ids.to(torch.int).argmax(dim=-1)
]
Since the CLIP model uses the token with the largest ID when calculating pooled_output
, the eos token must be the last token.