Cephalo Gemma 3 4b It 04 15 2025

🚀 Cephalo - Gemma - 3 - 4b

Cephalo - Gemma - 3 - 4b是一个多模态视觉语言模型，可用于生物启发材料的分析和设计。通过输入图像和文本提示，它能对图像内容进行详细分析，并以文本或JSON格式输出结果。

🚀 快速开始

加载模型

import torch
from transformers import AutoProcessor, Gemma3ForConditionalGeneration
from transformers.image_utils import load_image
from PIL import Image as PILImage

ckpt = "lamm-mit/Cephalo-Gemma-3-4b-it-04-15-2025"
model = Gemma3ForConditionalGeneration.from_pretrained(
    ckpt, device_map="auto", torch_dtype=torch.bfloat16,
)
processor = AutoProcessor.from_pretrained(ckpt)

💻 使用示例

基础用法

image=PILImage.open(f'./spiderweb.png').convert("RGB")
messages = [
    {
        "role": "system",
        "content": [
             {"type": "text", "text": "You are a materials scientist."}
        ],
        "role": "user",
        "content": [
             {"type": "image", "image": image},
             {"type": "text", "text": "What does this image show? Provide a detailed analysis."}
        ]
    }
]
inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt"
).to(model.device)

input_len = inputs["input_ids"].shape[-1]

generation = model.generate(**inputs, max_new_tokens=512, do_sample=False)
generation = generation[0][input_len:]

decoded = processor.decode(generation, skip_special_tokens=True)
print(decoded)

输出:

The image shows a comparison between a 3D model of a structure and its physical 3D printed counterpart. 

The top part of the image displays a 3D model of a structure, which is a complex geometric design with multiple interconnected lines and angles. The model is likely created using computer-aided design (CAD) software, which allows for precise and detailed representation of the structure.

The bottom part of the image shows the physical 3D printed version of the same structure. The printed object is a tangible representation of the CAD model, with the same geometric design and intricate details. The printed object is placed on a surface, which could be a table or a platform, and is illuminated to highlight its three-dimensional form.

The comparison between the 3D model and the physical 3D printed object demonstrates the accuracy and fidelity of the 3D printing process. The printed object closely resembles the CAD model, indicating that the 3D printing technology can accurately reproduce complex geometric designs.

The results shown in the image highlight the potential of 3D printing for creating complex and intricate structures with high precision and accuracy. This technology has various applications in fields such as manufacturing, engineering, and design, where the ability to create precise and detailed objects is crucial.

高级用法

image=PILImage.open(f'./scan.jpg').convert("RGB")
messages = [
    {
        "role": "system",
        "content": [
             {"type": "text", "text": "You are a biologist."}
        ],
        "role": "user",
        "content": [
             {"type": "image", "image": image},
             {"type": "text", "text": "Extract details of the material, type, design features, and use cases. Respond in JSON."}
        ]
    }
]
inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt"
).to(model.device)

input_len = inputs["input_ids"].shape[-1]

generation = model.generate(**inputs, max_new_tokens=512, do_sample=False)
generation = generation[0][input_len:]

decoded = processor.decode(generation, skip_special_tokens=True)
print(decoded)

输出:

Here's a detailed description of the material, type, design features, and use cases, presented in JSON format:

{
  "material": "Spider silk",
  "type": "Natural protein fiber",
  "design_features": [
    "High tensile strength and elasticity",
    "Lightweight and flexible",
    "Self-healing properties",
    "Biocompatible and biodegradable"
  ],
  "use_cases": [
    "Medical applications: sutures, scaffolds for tissue engineering, drug delivery systems",
    "Aerospace: lightweight composites for aircraft and spacecraft",
    "Protective gear: bulletproof vests, helmets, and body armor",
    "Industrial applications: high-performance fibers for ropes, cables, and nets",
    "Environmental applications: biodegradable packaging and textiles"
  ],
  "properties": {
    "tensile_strength": "Up to 1.5 GPa",
    "elastic_modulus": "1.5 GPa",
    "elongation_at_break": "Up to 1500%"
  },
  "source": "Spider silk is produced by spiders and is composed of proteins such as spidroin, which are arranged in a hierarchical structure to form the silk fibers."
}

**JSON字段说明**:

- **material**: 使用的主要材料，即蜘蛛丝。
- **type**: 材料类型，为天然蛋白质纤维。
- **design_features**: 材料的关键特性，包括高强度、弹性和自愈特性。
- **use_cases**: 蜘蛛丝的各种应用，从医疗到工业领域。
- **properties**: 蜘蛛丝的物理特性，如拉伸强度、弹性模量和断裂伸长率。
- **source**: 简要描述蜘蛛丝的来源及其组成。

此JSON提供了蜘蛛丝的全面概述，突出了其独特的特性和潜在应用。

📚 详细文档

引用

@article{Buehler_Cephalo_2024_journal,
  title={Cephalo: Multi-Modal Vision-Language Models for Bio-Inspired Materials Analysis and Design},
  author={Markus J. Buehler},
  journal={Advanced Functional Materials},
  year={2024},
  volume={34},
  issue={49},
  doi={2409531},
  url={https://advanced.onlinelibrary.wiley.com/doi/full/10.1002/adfm.202409531}
}