Llm Jp Clip Vit Large Patch14

🚀 llm-jp-clip-vit-large-patch14模型

本項目是一個日語CLIP模型，使用OpenCLIP在大規模日語圖像文本對上進行訓練，可用於零樣本圖像分類和圖像文本檢索等視覺語言任務。

🚀 快速開始

安裝

$ pip install open_clip_torch

零樣本圖像分類

import open_clip

model, preprocess = open_clip.create_model_from_pretrained('hf-hub:llm-jp/llm-jp-clip-vit-large-patch14')
tokenizer = open_clip.get_tokenizer('hf-hub:llm-jp/llm-jp-clip-vit-large-patch14')

import torch
from PIL import Image
import requests

url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
image = Image.open(requests.get(url, stream=True).raw)
image = preprocess(image).unsqueeze(0)
text = tokenizer(["貓", "犬", "鳥"])

with torch.no_grad(), torch.cuda.amp.autocast():
    image_features = model.encode_image(image)
    text_features = model.encode_text(text)
    image_features /= image_features.norm(dim=-1, keepdim=True)
    text_features /= text_features.norm(dim=-1, keepdim=True)

    text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)

print("Label probs:", text_probs)
# Label probs: tensor([[9.9425e-01, 5.2273e-03, 5.2600e-04]])

參考資料：

• 在Hugging Face使用OpenCLIP，HuggingFace文檔
• OpenCLIP 倉庫

✨ 主要特性

• 基於OpenCLIP訓練，可用於零樣本圖像分類和圖像文本檢索任務。
• 在大規模日語圖像文本對上進行訓練，參數總量達4.67億。

📦 安裝指南

$ pip install open_clip_torch

💻 使用示例

基礎用法

import open_clip

model, preprocess = open_clip.create_model_from_pretrained('hf-hub:llm-jp/llm-jp-clip-vit-large-patch14')
tokenizer = open_clip.get_tokenizer('hf-hub:llm-jp/llm-jp-clip-vit-large-patch14')

import torch
from PIL import Image
import requests

url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
image = Image.open(requests.get(url, stream=True).raw)
image = preprocess(image).unsqueeze(0)
text = tokenizer(["貓", "犬", "鳥"])

with torch.no_grad(), torch.cuda.amp.autocast():
    image_features = model.encode_image(image)
    text_features = model.encode_text(text)
    image_features /= image_features.norm(dim=-1, keepdim=True)
    text_features /= text_features.norm(dim=-1, keepdim=True)

    text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)

print("Label probs:", text_probs)
# Label probs: tensor([[9.9425e-01, 5.2273e-03, 5.2600e-04]])

📚 詳細文檔

模型詳情

該日語CLIP模型使用OpenCLIP在relaion2B-en-research-safe-japanese-translation上進行訓練，這是ReLAION - 5B英文子集的日語翻譯版本，由gemma - 2 - 9b - it翻譯。模型的總參數數量為4.67億。

訓練詳情

模型架構

• 文本編碼器：使用llm - jp - tokenizer的RoBERTa base
• 圖像編碼器：ViT - L/14

訓練數據

該模型在relaion2B-en-research-safe-japanese-translation上進行訓練。由於圖像下載成功率為70%，數據集大小為14.5億個樣本，共進行了9個epoch的處理（總共130億個樣本）。

評估

評估代碼：https://github.com/llm-jp/clip-eval

表格：各模型在零樣本圖像分類和圖像文本檢索任務中的性能表現。粗體表示第一名，_下劃線_表示第二名。

🔧 技術細節

• 模型使用OpenCLIP框架進行訓練，結合了文本編碼器和圖像編碼器，能夠學習圖像和文本之間的關聯。
• 通過在大規模日語圖像文本對上進行訓練，模型能夠在零樣本圖像分類和圖像文本檢索任務中取得較好的性能。

📄 許可證

Apache許可證，版本2.0

由於訓練數據使用gemma - 2 - 9b - it進行翻譯，請參考Gemma使用條款。我們僅將Gemma用於翻譯目的。根據第1.1(e)節中“模型衍生作品”的定義，我們的模型不屬於“為了使該模型表現得與Gemma相似的模型”類別。因此，我們認為沒有必要繼承Gemma許可證。

引用

@inproceedings{sugiura-etal-2025-developing,
    title = "Developing {J}apanese {CLIP} Models Leveraging an Open-weight {LLM} for Large-scale Dataset Translation",
    author = "Sugiura, Issa  and
      Kurita, Shuhei  and
      Oda, Yusuke  and
      Kawahara, Daisuke  and
      Okazaki, Naoaki",
    editor = "Ebrahimi, Abteen  and
      Haider, Samar  and
      Liu, Emmy  and
      Haider, Sammar  and
      Leonor Pacheco, Maria  and
      Wein, Shira",
    booktitle = "Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 4: Student Research Workshop)",
    month = apr,
    year = "2025",
    address = "Albuquerque, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.naacl-srw.15/",
    pages = "162--170",
    ISBN = "979-8-89176-192-6",
    abstract = "CLIP is a foundational model that bridges images and text, widely adopted as a key component in numerous vision-language models.However, the lack of large-scale open Japanese image-text pairs poses a significant barrier to the development of Japanese vision-language models.In this study, we constructed a Japanese image-text pair dataset with 1.5 billion examples using machine translation with open-weight LLMs and pre-trained Japanese CLIP models on the dataset.The performance of the pre-trained models was evaluated across seven benchmark datasets, achieving competitive average scores compared to models of similar size without the need for extensive data curation. However, the results also revealed relatively low performance on tasks specific to Japanese culture, highlighting the limitations of translation-based approaches in capturing cultural nuances. Our dataset, models, and code are publicly available."
}