clip-vit-large-patch14-ko Open-source Multimodal Model - Supports Bilingual Korean and English Understanding and Application

Home

Clip Vit Large Patch14 Ko

Developed by Bingsu

Korean CLIP model trained via knowledge distillation, supporting Korean and English multimodal understanding

Text-to-Image

Transformers

KoreanOpen Source License:MIT #Korean CLIP #Zero-shot Image Classification #Knowledge Distillation

Downloads 4,537

Release Time : 10/11/2022

Model Overview

This is a Korean CLIP model trained using knowledge distillation methods, capable of understanding the relationship between Korean text and images, suitable for multimodal tasks.

Model Features

Korean Optimization

Specially optimized for Korean, demonstrating excellent performance in Korean multimodal tasks

Knowledge Distillation

Utilizes knowledge distillation to transfer learning from English CLIP models, preserving the original model's powerful capabilities

Multimodal Understanding

Capable of simultaneously understanding the relationship between image content and Korean text descriptions

Model Capabilities

Zero-shot image classification

Image-text matching

Cross-modal retrieval

Multimodal understanding

Use Cases

Content Moderation

Korean Content Classification

Automatic classification of Korean social media content

E-commerce

Product Image Search

Search for relevant product images using Korean descriptions

🚀 clip-vit-large-patch14-ko

A Korean CLIP model trained by Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation. It aims to provide effective image - text matching capabilities in the Korean language.

🚀 Quick Start

This is a Korean CLIP model trained based on the method described in Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation.

Training code: https://github.com/Bing-su/KoCLIP_training_code

Used data: All Korean - English parallel data in AIHUB

Widget Information

src: https://huggingface.co/datasets/mishig/sample_images/resolve/main/cat-dog-music.png
candidate_labels: Playing guitar cat, Playing piano dog
example_title: Guitar, cat and dog

License

This project is licensed under the MIT license.

💻 Usage Examples

Basic Usage

import requests
import torch
from PIL import Image
from transformers import AutoModel, AutoProcessor

repo = "Bingsu/clip-vit-large-patch14-ko"
model = AutoModel.from_pretrained(repo)
processor = AutoProcessor.from_pretrained(repo)

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)
inputs = processor(text=["Two cats", "Two dogs"], images=image, return_tensors="pt", padding=True)
with torch.inference_mode():
    outputs = model(**inputs)
logits_per_image = outputs.logits_per_image
probs = logits_per_image.softmax(dim=1)

>>> probs
tensor([[0.9974, 0.0026]])

Advanced Usage

from transformers import pipeline

repo = "Bingsu/clip-vit-large-patch14-ko"
pipe = pipeline("zero-shot-image-classification", model=repo)

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
result = pipe(images=url, candidate_labels=["One cat", "Two cats", "Friends of cats lying on a pink sofa"], hypothesis_template="{}")

>>> result
[{'score': 0.9907576441764832, 'label': 'Friends of cats lying on a pink sofa'},
 {'score': 0.009206341579556465, 'label': 'Two cats'},
 {'score': 3.606083555496298e-05, 'label': 'One cat'}]

📄 License

This project is released under the MIT license.

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご