Cephalo Gemma 3 4b It 04 15 2025

🚀 Cephalo - Gemma - 3 - 4b

Cephalo - Gemma - 3 - 4b是一個多模態視覺語言模型，可用於生物啟發材料的分析和設計。通過輸入圖像和文本提示，它能對圖像內容進行詳細分析，並以文本或JSON格式輸出結果。

🚀 快速開始

加載模型

import torch
from transformers import AutoProcessor, Gemma3ForConditionalGeneration
from transformers.image_utils import load_image
from PIL import Image as PILImage

ckpt = "lamm-mit/Cephalo-Gemma-3-4b-it-04-15-2025"
model = Gemma3ForConditionalGeneration.from_pretrained(
    ckpt, device_map="auto", torch_dtype=torch.bfloat16,
)
processor = AutoProcessor.from_pretrained(ckpt)

💻 使用示例

基礎用法

image=PILImage.open(f'./spiderweb.png').convert("RGB")
messages = [
    {
        "role": "system",
        "content": [
             {"type": "text", "text": "You are a materials scientist."}
        ],
        "role": "user",
        "content": [
             {"type": "image", "image": image},
             {"type": "text", "text": "What does this image show? Provide a detailed analysis."}
        ]
    }
]
inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt"
).to(model.device)

input_len = inputs["input_ids"].shape[-1]

generation = model.generate(**inputs, max_new_tokens=512, do_sample=False)
generation = generation[0][input_len:]

decoded = processor.decode(generation, skip_special_tokens=True)
print(decoded)

輸出:

The image shows a comparison between a 3D model of a structure and its physical 3D printed counterpart. 

The top part of the image displays a 3D model of a structure, which is a complex geometric design with multiple interconnected lines and angles. The model is likely created using computer-aided design (CAD) software, which allows for precise and detailed representation of the structure.

The bottom part of the image shows the physical 3D printed version of the same structure. The printed object is a tangible representation of the CAD model, with the same geometric design and intricate details. The printed object is placed on a surface, which could be a table or a platform, and is illuminated to highlight its three-dimensional form.

The comparison between the 3D model and the physical 3D printed object demonstrates the accuracy and fidelity of the 3D printing process. The printed object closely resembles the CAD model, indicating that the 3D printing technology can accurately reproduce complex geometric designs.

The results shown in the image highlight the potential of 3D printing for creating complex and intricate structures with high precision and accuracy. This technology has various applications in fields such as manufacturing, engineering, and design, where the ability to create precise and detailed objects is crucial.

高級用法

image=PILImage.open(f'./scan.jpg').convert("RGB")
messages = [
    {
        "role": "system",
        "content": [
             {"type": "text", "text": "You are a biologist."}
        ],
        "role": "user",
        "content": [
             {"type": "image", "image": image},
             {"type": "text", "text": "Extract details of the material, type, design features, and use cases. Respond in JSON."}
        ]
    }
]
inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt"
).to(model.device)

input_len = inputs["input_ids"].shape[-1]

generation = model.generate(**inputs, max_new_tokens=512, do_sample=False)
generation = generation[0][input_len:]

decoded = processor.decode(generation, skip_special_tokens=True)
print(decoded)

輸出:

Here's a detailed description of the material, type, design features, and use cases, presented in JSON format:

{
  "material": "Spider silk",
  "type": "Natural protein fiber",
  "design_features": [
    "High tensile strength and elasticity",
    "Lightweight and flexible",
    "Self-healing properties",
    "Biocompatible and biodegradable"
  ],
  "use_cases": [
    "Medical applications: sutures, scaffolds for tissue engineering, drug delivery systems",
    "Aerospace: lightweight composites for aircraft and spacecraft",
    "Protective gear: bulletproof vests, helmets, and body armor",
    "Industrial applications: high-performance fibers for ropes, cables, and nets",
    "Environmental applications: biodegradable packaging and textiles"
  ],
  "properties": {
    "tensile_strength": "Up to 1.5 GPa",
    "elastic_modulus": "1.5 GPa",
    "elongation_at_break": "Up to 1500%"
  },
  "source": "Spider silk is produced by spiders and is composed of proteins such as spidroin, which are arranged in a hierarchical structure to form the silk fibers."
}

**JSON字段說明**:

- **material**: 使用的主要材料，即蜘蛛絲。
- **type**: 材料類型，為天然蛋白質纖維。
- **design_features**: 材料的關鍵特性，包括高強度、彈性和自愈特性。
- **use_cases**: 蜘蛛絲的各種應用，從醫療到工業領域。
- **properties**: 蜘蛛絲的物理特性，如拉伸強度、彈性模量和斷裂伸長率。
- **source**: 簡要描述蜘蛛絲的來源及其組成。

此JSON提供了蜘蛛絲的全面概述，突出了其獨特的特性和潛在應用。

📚 詳細文檔

引用

@article{Buehler_Cephalo_2024_journal,
  title={Cephalo: Multi-Modal Vision-Language Models for Bio-Inspired Materials Analysis and Design},
  author={Markus J. Buehler},
  journal={Advanced Functional Materials},
  year={2024},
  volume={34},
  issue={49},
  doi={2409531},
  url={https://advanced.onlinelibrary.wiley.com/doi/full/10.1002/adfm.202409531}
}