Cephalo Gemma 3 4b It 04 15 2025

🚀 Cephalo - Gemma - 3 - 4b

Cephalo - Gemma - 3 - 4bは、画像とテキストを組み合わせた多モーダルなモデルで、材料科学や生物学などの分野における画像解析に役立ちます。

🚀 クイックスタート

モデルの読み込み

import torch
from transformers import AutoProcessor, Gemma3ForConditionalGeneration
from transformers.image_utils import load_image
from PIL import Image as PILImage

ckpt = "lamm-mit/Cephalo-Gemma-3-4b-it-04-15-2025"
model = Gemma3ForConditionalGeneration.from_pretrained(
    ckpt, device_map="auto", torch_dtype=torch.bfloat16,
)
processor = AutoProcessor.from_pretrained(ckpt)

💻 使用例

基本的な使用法

image=PILImage.open(f'./spiderweb.png').convert("RGB")
messages = [
    {
        "role": "system",
        "content": [
             {"type": "text", "text": "You are a materials scientist."}
        ],
        "role": "user",
        "content": [
             {"type": "image", "image": image},
             {"type": "text", "text": "What does this image show? Provide a detailed analysis."}
        ]
    }
]
inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt"
).to(model.device)

input_len = inputs["input_ids"].shape[-1]

generation = model.generate(**inputs, max_new_tokens=512, do_sample=False)
generation = generation[0][input_len:]

decoded = processor.decode(generation, skip_special_tokens=True)
print(decoded)

出力:

The image shows a comparison between a 3D model of a structure and its physical 3D printed counterpart. 

The top part of the image displays a 3D model of a structure, which is a complex geometric design with multiple interconnected lines and angles. The model is likely created using computer-aided design (CAD) software, which allows for precise and detailed representation of the structure.

The bottom part of the image shows the physical 3D printed version of the same structure. The printed object is a tangible representation of the CAD model, with the same geometric design and intricate details. The printed object is placed on a surface, which could be a table or a platform, and is illuminated to highlight its three-dimensional form.

The comparison between the 3D model and the physical 3D printed object demonstrates the accuracy and fidelity of the 3D printing process. The printed object closely resembles the CAD model, indicating that the 3D printing technology can accurately reproduce complex geometric designs.

The results shown in the image highlight the potential of 3D printing for creating complex and intricate structures with high precision and accuracy. This technology has various applications in fields such as manufacturing, engineering, and design, where the ability to create precise and detailed objects is crucial.

高度な使用法

image=PILImage.open(f'./scan.jpg').convert("RGB")
messages = [
    {
        "role": "system",
        "content": [
             {"type": "text", "text": "You are a biologist."}
        ],
        "role": "user",
        "content": [
             {"type": "image", "image": image},
             {"type": "text", "text": "Extract details of the material, type, design features, and use cases. Respond in JSON."}
        ]
    }
]
inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt"
).to(model.device)

input_len = inputs["input_ids"].shape[-1]

generation = model.generate(**inputs, max_new_tokens=512, do_sample=False)
generation = generation[0][input_len:]

decoded = processor.decode(generation, skip_special_tokens=True)
print(decoded)

出力:

Here's a detailed description of the material, type, design features, and use cases, presented in JSON format:

{
  "material": "Spider silk",
  "type": "Natural protein fiber",
  "design_features": [
    "High tensile strength and elasticity",
    "Lightweight and flexible",
    "Self-healing properties",
    "Biocompatible and biodegradable"
  ],
  "use_cases": [
    "Medical applications: sutures, scaffolds for tissue engineering, drug delivery systems",
    "Aerospace: lightweight composites for aircraft and spacecraft",
    "Protective gear: bulletproof vests, helmets, and body armor",
    "Industrial applications: high-performance fibers for ropes, cables, and nets",
    "Environmental applications: biodegradable packaging and textiles"
  ],
  "properties": {
    "tensile_strength": "Up to 1.5 GPa",
    "elastic_modulus": "1.5 GPa",
    "elongation_at_break": "Up to 1500%"
  },
  "source": "Spider silk is produced by spiders and is composed of proteins such as spidroin, which are arranged in a hierarchical structure to form the silk fibers."
}

JSONフィールドの説明:

• material: 使用される主要な材料で、ここでは蜘蛛の糸です。
• type: 材料の種類で、天然のタンパク質繊維です。
• design_features: 材料の主要な特性で、強度、弾性、自己修復特性などが含まれます。
• use_cases: 蜘蛛の糸が使用される様々なアプリケーションで、医療から産業まで幅広い分野があります。
• properties: 蜘蛛の糸の物理的特性で、引張強度、弾性率、破断点伸びなどが含まれます。
• source: 蜘蛛の糸の由来とその構成についての簡単な説明です。

このJSONは、蜘蛛の糸の独自の特性と潜在的なアプリケーションを包括的に概説しています。

📚 参考文献

@article{Buehler_Cephalo_2024_journal,
  title={Cephalo: Multi-Modal Vision-Language Models for Bio-Inspired Materials Analysis and Design},
  author={Markus J. Buehler},
  journal={Advanced Functional Materials},
  year={2024},
  volume={34},
  issue={49},
  doi={2409531},
  url={https://advanced.onlinelibrary.wiley.com/doi/full/10.1002/adfm.202409531}
}