Japanese Clip Vit B 32 Roberta Base

🚀 recruit-jp/japanese-clip-vit-b-32-roberta-base

本项目是一个面向日语的对比语言 - 图像预训练模型，能够将日语文本和图像映射到同一嵌入空间，可用于零样本图像分类、文本 - 图像检索、图像特征提取等任务。

🚀 快速开始

安装依赖

pip install pillow requests transformers torch torchvision sentencepiece

运行示例代码

import io
import requests

import torch
import torchvision
from PIL import Image
from transformers import AutoTokenizer, AutoModel


model_name = "recruit-jp/japanese-clip-vit-b-32-roberta-base"
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name, trust_remote_code=True).to(device)


def _convert_to_rgb(image):
    return image.convert('RGB')


preprocess = torchvision.transforms.Compose([
    torchvision.transforms.Resize(size=224, interpolation=torchvision.transforms.InterpolationMode.BICUBIC, max_size=None),
    torchvision.transforms.CenterCrop(size=(224, 224)),
    _convert_to_rgb,
    torchvision.transforms.ToTensor(),
    torchvision.transforms.Normalize(mean=[0.48145466, 0.4578275, 0.40821073], std=[0.26862954, 0.26130258, 0.27577711])
])


def tokenize(tokenizer, texts):
    texts = ["[CLS]" + text for text in texts]
    encodings = [
        # NOTE: the maximum token length that can be fed into this model is 77
        tokenizer(text, max_length=77, padding="max_length", truncation=True, add_special_tokens=False)["input_ids"]
        for text in texts
    ]
    return torch.LongTensor(encodings)


# Run!
image = Image.open(
    io.BytesIO(
        requests.get(
            'https://images.pexels.com/photos/2253275/pexels-photo-2253275.jpeg?auto=compress&cs=tinysrgb&dpr=3&h=750&w=1260'
        ).content
    )
)
image = preprocess(image).unsqueeze(0).to(device)
text = tokenize(tokenizer, texts=["犬", "猫", "象"]).to(device)
with torch.inference_mode():
    image_features = model.get_image_features(image)
    image_features /= image_features.norm(dim=-1, keepdim=True)
    text_features = model.get_text_features(input_ids=text)
    text_features /= text_features.norm(dim=-1, keepdim=True)
    probs = image_features @ text_features.T
print("Label probs:", probs.cpu().numpy()[0])

✨ 主要特性

• 语言适配：专为日语设计，使用日语子集的 LAION2B-multi 数据集进行训练。
• 多任务支持：可用于零样本图像分类、文本 - 图像检索、图像特征提取等多种任务。
• 模型架构：图像编码器采用 laion/CLIP-ViT-B-32-laion2B-s34B-b79K，文本编码器采用 rinna/japanese-roberta-base。

📦 安装指南

安装所需的 Python 包：

pip install pillow requests transformers torch torchvision sentencepiece

💻 使用示例

基础用法

import io
import requests

import torch
import torchvision
from PIL import Image
from transformers import AutoTokenizer, AutoModel


model_name = "recruit-jp/japanese-clip-vit-b-32-roberta-base"
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name, trust_remote_code=True).to(device)


def _convert_to_rgb(image):
    return image.convert('RGB')


preprocess = torchvision.transforms.Compose([
    torchvision.transforms.Resize(size=224, interpolation=torchvision.transforms.InterpolationMode.BICUBIC, max_size=None),
    torchvision.transforms.CenterCrop(size=(224, 224)),
    _convert_to_rgb,
    torchvision.transforms.ToTensor(),
    torchvision.transforms.Normalize(mean=[0.48145466, 0.4578275, 0.40821073], std=[0.26862954, 0.26130258, 0.27577711])
])


def tokenize(tokenizer, texts):
    texts = ["[CLS]" + text for text in texts]
    encodings = [
        # NOTE: the maximum token length that can be fed into this model is 77
        tokenizer(text, max_length=77, padding="max_length", truncation=True, add_special_tokens=False)["input_ids"]
        for text in texts
    ]
    return torch.LongTensor(encodings)


# Run!
image = Image.open(
    io.BytesIO(
        requests.get(
            'https://images.pexels.com/photos/2253275/pexels-photo-2253275.jpeg?auto=compress&cs=tinysrgb&dpr=3&h=750&w=1260'
        ).content
    )
)
image = preprocess(image).unsqueeze(0).to(device)
text = tokenize(tokenizer, texts=["犬", "猫", "象"]).to(device)
with torch.inference_mode():
    image_features = model.get_image_features(image)
    image_features /= image_features.norm(dim=-1, keepdim=True)
    text_features = model.get_text_features(input_ids=text)
    text_features /= text_features.norm(dim=-1, keepdim=True)
    probs = image_features @ text_features.T
print("Label probs:", probs.cpu().numpy()[0])

📚 详细文档

模型概述

• 开发者：Recruit Co., Ltd.
• 模型类型：对比语言 - 图像预训练模型
• 支持语言：日语
• 许可证：CC-BY-4.0

更多详细信息请参考技术博客文章。

模型性能

在以下数据集上进行了模型性能评估。由于 ImageNet V2 和 Food101 是英语语境下的数据集，在评估前将类别标签翻译成了日语。

训练数据集

该模型使用 LAION2B-multi 数据集的日语子集的 1.28 亿个图像 - 文本对进行训练。

免责声明

Recruit Co., Ltd. 对本模型使用的结果不进行准确性、有用性、确定性、合法性的确认，也不提供任何保证和补偿。对于因使用本模型而给用户造成的损害以及与第三方之间的纠纷，Recruit Co., Ltd. 不承担任何责任。

📄 许可证

本模型采用 CC-BY-4.0 许可证。