Llm Jp Clip Vit Large Patch14

🚀 llm-jp-clip-vit-large-patch14模型

本项目是一个日语CLIP模型，使用OpenCLIP在大规模日语图像文本对上进行训练，可用于零样本图像分类和图像文本检索等视觉语言任务。

🚀 快速开始

安装

$ pip install open_clip_torch

零样本图像分类

import open_clip

model, preprocess = open_clip.create_model_from_pretrained('hf-hub:llm-jp/llm-jp-clip-vit-large-patch14')
tokenizer = open_clip.get_tokenizer('hf-hub:llm-jp/llm-jp-clip-vit-large-patch14')

import torch
from PIL import Image
import requests

url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
image = Image.open(requests.get(url, stream=True).raw)
image = preprocess(image).unsqueeze(0)
text = tokenizer(["猫", "犬", "鳥"])

with torch.no_grad(), torch.cuda.amp.autocast():
    image_features = model.encode_image(image)
    text_features = model.encode_text(text)
    image_features /= image_features.norm(dim=-1, keepdim=True)
    text_features /= text_features.norm(dim=-1, keepdim=True)

    text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)

print("Label probs:", text_probs)
# Label probs: tensor([[9.9425e-01, 5.2273e-03, 5.2600e-04]])

参考资料：

• 在Hugging Face使用OpenCLIP，HuggingFace文档
• OpenCLIP 仓库

✨ 主要特性

• 基于OpenCLIP训练，可用于零样本图像分类和图像文本检索任务。
• 在大规模日语图像文本对上进行训练，参数总量达4.67亿。

📦 安装指南

$ pip install open_clip_torch

💻 使用示例

基础用法

import open_clip

model, preprocess = open_clip.create_model_from_pretrained('hf-hub:llm-jp/llm-jp-clip-vit-large-patch14')
tokenizer = open_clip.get_tokenizer('hf-hub:llm-jp/llm-jp-clip-vit-large-patch14')

import torch
from PIL import Image
import requests

url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
image = Image.open(requests.get(url, stream=True).raw)
image = preprocess(image).unsqueeze(0)
text = tokenizer(["猫", "犬", "鳥"])

with torch.no_grad(), torch.cuda.amp.autocast():
    image_features = model.encode_image(image)
    text_features = model.encode_text(text)
    image_features /= image_features.norm(dim=-1, keepdim=True)
    text_features /= text_features.norm(dim=-1, keepdim=True)

    text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)

print("Label probs:", text_probs)
# Label probs: tensor([[9.9425e-01, 5.2273e-03, 5.2600e-04]])

📚 详细文档

模型详情

该日语CLIP模型使用OpenCLIP在relaion2B-en-research-safe-japanese-translation上进行训练，这是ReLAION - 5B英文子集的日语翻译版本，由gemma - 2 - 9b - it翻译。模型的总参数数量为4.67亿。

训练详情

模型架构

• 文本编码器：使用llm - jp - tokenizer的RoBERTa base
• 图像编码器：ViT - L/14

训练数据

该模型在relaion2B-en-research-safe-japanese-translation上进行训练。由于图像下载成功率为70%，数据集大小为14.5亿个样本，共进行了9个epoch的处理（总共130亿个样本）。

评估

评估代码：https://github.com/llm-jp/clip-eval

表格：各模型在零样本图像分类和图像文本检索任务中的性能表现。粗体表示第一名，_下划线_表示第二名。

🔧 技术细节

• 模型使用OpenCLIP框架进行训练，结合了文本编码器和图像编码器，能够学习图像和文本之间的关联。
• 通过在大规模日语图像文本对上进行训练，模型能够在零样本图像分类和图像文本检索任务中取得较好的性能。

📄 许可证

Apache许可证，版本2.0

由于训练数据使用gemma - 2 - 9b - it进行翻译，请参考Gemma使用条款。我们仅将Gemma用于翻译目的。根据第1.1(e)节中“模型衍生作品”的定义，我们的模型不属于“为了使该模型表现得与Gemma相似的模型”类别。因此，我们认为没有必要继承Gemma许可证。

引用

@inproceedings{sugiura-etal-2025-developing,
    title = "Developing {J}apanese {CLIP} Models Leveraging an Open-weight {LLM} for Large-scale Dataset Translation",
    author = "Sugiura, Issa  and
      Kurita, Shuhei  and
      Oda, Yusuke  and
      Kawahara, Daisuke  and
      Okazaki, Naoaki",
    editor = "Ebrahimi, Abteen  and
      Haider, Samar  and
      Liu, Emmy  and
      Haider, Sammar  and
      Leonor Pacheco, Maria  and
      Wein, Shira",
    booktitle = "Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 4: Student Research Workshop)",
    month = apr,
    year = "2025",
    address = "Albuquerque, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.naacl-srw.15/",
    pages = "162--170",
    ISBN = "979-8-89176-192-6",
    abstract = "CLIP is a foundational model that bridges images and text, widely adopted as a key component in numerous vision-language models.However, the lack of large-scale open Japanese image-text pairs poses a significant barrier to the development of Japanese vision-language models.In this study, we constructed a Japanese image-text pair dataset with 1.5 billion examples using machine translation with open-weight LLMs and pre-trained Japanese CLIP models on the dataset.The performance of the pre-trained models was evaluated across seven benchmark datasets, achieving competitive average scores compared to models of similar size without the need for extensive data curation. However, the results also revealed relatively low performance on tasks specific to Japanese culture, highlighting the limitations of translation-based approaches in capturing cultural nuances. Our dataset, models, and code are publicly available."
}