Llm Jp Clip Vit Base Patch16

🚀 llm-jp-clip-vit-base-patch16模型

本项目是基于OpenCLIP训练的日语CLIP模型，利用大规模日语图像文本对数据集进行训练，可用于零样本图像分类等视觉语言任务，为日语视觉语言处理提供了有效的解决方案。

🚀 快速开始

安装

$ pip install open_clip_torch

零样本图像分类示例

import open_clip

model, preprocess = open_clip.create_model_from_pretrained('hf-hub:llm-jp/llm-jp-clip-vit-base-patch16')
tokenizer = open_clip.get_tokenizer('hf-hub:llm-jp/llm-jp-clip-vit-base-patch16')

import torch
from PIL import Image
import requests

url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
image = Image.open(requests.get(url, stream=True).raw)
image = preprocess(image).unsqueeze(0)
text = tokenizer(["猫", "犬", "鳥"])

with torch.no_grad(), torch.cuda.amp.autocast():
    image_features = model.encode_image(image)
    text_features = model.encode_text(text)
    image_features /= image_features.norm(dim=-1, keepdim=True)
    text_features /= text_features.norm(dim=-1, keepdim=True)

    text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)

print("Label probs:", text_probs)
# Label probs: tensor([[9.9425e-01, 5.2273e-03, 5.2600e-04]])

参考资料：

• 在Hugging Face上使用OpenCLIP，HuggingFace文档
• OpenCLIP 仓库

✨ 主要特性

• 基于OpenCLIP训练的日语CLIP模型。
• 使用relaion2B-en-research-safe-japanese-translation数据集进行训练。
• 模型总参数数量为2.48亿。

📦 安装指南

$ pip install open_clip_torch

💻 使用示例

基础用法

import open_clip

model, preprocess = open_clip.create_model_from_pretrained('hf-hub:llm-jp/llm-jp-clip-vit-base-patch16')
tokenizer = open_clip.get_tokenizer('hf-hub:llm-jp/llm-jp-clip-vit-base-patch16')

import torch
from PIL import Image
import requests

url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
image = Image.open(requests.get(url, stream=True).raw)
image = preprocess(image).unsqueeze(0)
text = tokenizer(["猫", "犬", "鳥"])

with torch.no_grad(), torch.cuda.amp.autocast():
    image_features = model.encode_image(image)
    text_features = model.encode_text(text)
    image_features /= image_features.norm(dim=-1, keepdim=True)
    text_features /= text_features.norm(dim=-1, keepdim=True)

    text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)

print("Label probs:", text_probs)
# Label probs: tensor([[9.9425e-01, 5.2273e-03, 5.2600e-04]])

📚 详细文档

模型详情

该日语CLIP模型使用OpenCLIP在relaion2B-en-research-safe-japanese-translation数据集上进行训练。该数据集是ReLAION - 5B英文子集（https://huggingface.co/datasets/laion/relaion2B-en-research-safe）的日语翻译版本，由gemma - 2 - 9b - it翻译。

该模型的总参数数量为2.48亿。

训练细节

模型架构

• 文本编码器：使用llm - jp - tokenizer的RoBERTa base。
• 图像编码器：ViT - B/16。

训练数据

该模型在relaion2B-en-research-safe-japanese-translation数据集上进行训练。由于图像下载成功率为70%，数据集大小为14.5亿个样本，共进行了9个轮次的训练（总共处理了130亿个样本）。

评估

评估代码：https://github.com/llm-jp/clip-eval

表格：各模型在零样本图像分类和图像文本检索任务中的性能表现。粗体表示第一名，_下划线_表示第二名。

📄 许可证

Apache许可证，版本2.0

请参考Gemma使用条款，因为训练数据使用gemma - 2 - 9b - it进行翻译。我们仅将Gemma用于翻译目的。根据第1.1(e)节中“模型衍生作品”的定义，我们的模型不属于“为使模型表现得与Gemma相似的模型”类别。因此，我们得出结论，无需继承Gemma许可证。

引用

@inproceedings{sugiura-etal-2025-developing,
    title = "Developing {J}apanese {CLIP} Models Leveraging an Open-weight {LLM} for Large-scale Dataset Translation",
    author = "Sugiura, Issa  and
      Kurita, Shuhei  and
      Oda, Yusuke  and
      Kawahara, Daisuke  and
      Okazaki, Naoaki",
    editor = "Ebrahimi, Abteen  and
      Haider, Samar  and
      Liu, Emmy  and
      Haider, Sammar  and
      Leonor Pacheco, Maria  and
      Wein, Shira",
    booktitle = "Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 4: Student Research Workshop)",
    month = apr,
    year = "2025",
    address = "Albuquerque, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.naacl-srw.15/",
    pages = "162--170",
    ISBN = "979-8-89176-192-6",
    abstract = "CLIP is a foundational model that bridges images and text, widely adopted as a key component in numerous vision-language models.However, the lack of large-scale open Japanese image-text pairs poses a significant barrier to the development of Japanese vision-language models.In this study, we constructed a Japanese image-text pair dataset with 1.5 billion examples using machine translation with open-weight LLMs and pre-trained Japanese CLIP models on the dataset.The performance of the pre-trained models was evaluated across seven benchmark datasets, achieving competitive average scores compared to models of similar size without the need for extensive data curation. However, the results also revealed relatively low performance on tasks specific to Japanese culture, highlighting the limitations of translation-based approaches in capturing cultural nuances. Our dataset, models, and code are publicly available."
}