Janus Pro 1B ONNX

🚀 ONNX版Janus - Pro - 1B模型

本項目將 https://huggingface.co/deepseek - ai/Janus - Pro - 1B 模型轉換為ONNX權重，以兼容Transformers.js庫。該模型支持多種跨模態任務，如文本到圖像、圖像到文本、圖像與文本結合到文本等。

🚀 快速開始

本項目使用 Transformers.js 這個JavaScript庫，你可以通過以下命令從 NPM 安裝：

npm i @huggingface/transformers

💻 使用示例

基礎用法

示例：圖像+文本到文本

import { AutoProcessor, MultiModalityCausalLM } from "@huggingface/transformers";

// Load processor and model
const model_id = "onnx-community/Janus-Pro-1B-ONNX";
const processor = await AutoProcessor.from_pretrained(model_id);
const model = await MultiModalityCausalLM.from_pretrained(model_id);

// Prepare inputs
const conversation = [
  {
    role: "<|User|>",
    content: "<image_placeholder>\nConvert the formula into latex code.",
    images: ["https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/quadratic_formula.png"],
  },
];
const inputs = await processor(conversation);

// Generate response
const outputs = await model.generate({
  ...inputs,
  max_new_tokens: 150,
  do_sample: false,
});

// Decode output
const new_tokens = outputs.slice(null, [inputs.input_ids.dims.at(-1), null]);
const decoded = processor.batch_decode(new_tokens, { skip_special_tokens: true });
console.log(decoded[0]);

示例：文本到圖像

import { AutoProcessor, MultiModalityCausalLM } from "@huggingface/transformers";

// Load processor and model
const model_id = "onnx-community/Janus-Pro-1B-ONNX";
const processor = await AutoProcessor.from_pretrained(model_id);
const model = await MultiModalityCausalLM.from_pretrained(model_id);

// Prepare inputs
const conversation = [
  {
    role: "<|User|>",
    content: "A stunning princess from kabul in red, white traditional clothing, blue eyes, brown hair",
  },
];
const inputs = await processor(conversation, { chat_template: "text_to_image" });

// Generate response
const num_image_tokens = processor.num_image_tokens;
const outputs = await model.generate_images({
  ...inputs,
  min_new_tokens: num_image_tokens,
  max_new_tokens: num_image_tokens,
  do_sample: true,
});

// Save the generated image
await outputs[0].save("test.png");

📄 許可證

本項目採用MIT許可證。

📦 模型信息