LegoGPT Open-Source AI System - Generate Physically Stable Lego Block Models for Free with Text Prompts

Legogpt

Developed by AvaLovelace

LegoGPT is the first AI system that generates physically stable LEGO brick models from text prompts, fine-tuned based on Llama-3.2-1B-Instruct.

Large Language Model

Safetensors

EnglishOpen Source License:MIT #LEGO 3D Generation #Physically Stable Structures #Text-to-Brick Design

Downloads 1,960

Release Time : 5/6/2025

Model Overview

This model can generate physically stable LEGO structures within a 20x20x20 grid based on text descriptions, supporting 21 common object categories.

Model Features

Physical Stability Guarantee

Generated LEGO designs meet physical stability requirements and can be directly assembled.

Category-Specific Optimization

Specialized optimization for 21 common object categories ensures higher generation quality.

Efficient Fine-tuning Technique

Uses LoRA for parameter-efficient fine-tuning, adjusting only q_proj and v_proj matrices.

Model Capabilities

Text-to-3D structure generation

LEGO design generation

Physical stability verification

Use Cases

Creative Design

Furniture Design

Generates buildable LEGO furniture models from text descriptions.

Can generate 21 types of furniture, including beds, chairs, and tables.

Educational Applications

Helps students quickly transform ideas into buildable LEGO models.

Simplifies the learning process of physical modeling.

Prototyping

Product Prototyping

Quickly generates LEGO prototypes for product concepts.

Accelerates the design iteration process.

🚀 LegoGPT

LegoGPT is the first model that can generate physically stable LEGO brick models from text prompts, offering a novel way to transform text into 3D LEGO designs.

🚀 Quick Start

See the GitHub repo for usage examples and an interactive CLI demo.

✨ Features

LegoGPT is the first approach for generating physically stable LEGO brick models from text prompts, as described in Generating Physically Stable and Buildable LEGO® Designs from Text.

📚 Documentation

Model Details

Model Description

Property	Details
Developed by	Carnegie Mellon University Generative Intelligence Lab
Funded by	This work is partly supported by the Packard Foundation, Cisco Research Grant, and Amazon Faculty Award. This work is also in part supported by the Manufacturing Futures Institute, Carnegie Mellon University, through a grant from the Richard King Mellon Foundation. KD is supported by the Microsoft Research PhD Fellowship.
Model Type	Autoregressive
Language(s)	English
License	MIT
Finetuned from model	meta-llama/Llama-3.2-1B-Instruct
Project page	https://avalovelace1.github.io/LegoGPT/

Model Sources

Property	Details
Repository	AvaLovelace1/LegoGPT
Paper	Generating Physically Stable and Buildable LEGO® Designs from Text
Demo	cmu-gil/LegoGPT-Demo

Limitations

The model is restricted to creating structures made of 1 - unit - tall cuboid bricks on a 20x20x20 grid. It was trained on a dataset of 21 object categories: basket, bed, bench, birdhouse, bookshelf, bottle, bowl, bus, camera, car, chair, guitar, jar, mug, piano, pot, sofa, table, tower, train, vessel. Performance on prompts from outside these categories may be limited.

Training Details

Training Data

LegoGPT was trained using StableText2Lego, a dataset of 47k LEGO structures.

Training Procedure

The model was fine - tuned using LoRA applied to the q_proj and v_proj matrices. We used AdamW optimization. The learning rate followed a cosine decay with warmup.

Training Hyperparameters

Property	Details
Training regime	bf16 mixed precision
Epochs	3
Global batch size	64
Max learning rate	0.002
Learning rate warmup steps	100
LoRA rank	32
LoRA alpha	16
LoRA dropout	0.05

Evaluation

See the paper for detailed evaluations.

Environmental Impact

Property	Details
Hardware Type	8x NVIDIA RTX A6000 (48 GB)
Hours used	0.5

Citation

If you find this model useful for your research, please cite the following work.

@article{pun2025legogpt,
    title   = {Generating Physically Stable and Buildable LEGO Designs from Text},
    author  = {Pun, Ava and Deng, Kangle and Liu, Ruixuan and Ramanan, Deva and Liu, Changliu and Zhu, Jun - Yan},
    journal = {arXiv preprint arXiv:2505.05469},
    year    = {2025}
}

Model Card Contact

Ava Pun (apun@andrew.cmu.edu)

Framework versions

PEFT 0.15.0

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご