Brickgpt

🚀 BrickGPT

BrickGPT is the first model for generating physically stable toy brick models from text prompts, offering a new way to create 3D brick structures.

🚀 Quick Start

See the GitHub repo for usage examples and an interactive CLI demo.

✨ Features

• Generate physically stable toy brick models from text prompts.
• Fine - tuned from a powerful base model.

📚 Documentation

Model Details

Model Description

Model Sources

Limitations

The model is restricted to creating structures made of 1 - unit - tall cuboid bricks on a 20x20x20 grid. It was trained on a dataset of 21 object categories: basket, bed, bench, birdhouse, bookshelf, bottle, bowl, bus, camera, car, chair, guitar, jar, mug, piano, pot, sofa, table, tower, train, vessel. Performance on prompts from outside these categories may be limited.

Training Details

Training Data

BrickGPT was trained using StableText2Brick, a dataset of 47k toy brick structures.

Training Procedure

The model was fine - tuned using LoRA applied to the q_proj and v_proj matrices. We used AdamW optimization. The learning rate followed a cosine decay with warmup.

Training Hyperparameters

Evaluation

See the paper for detailed evaluations.

Environmental Impact

Citation

If you find this model useful for your research, please cite the following work.

@article{pun2025brickgpt,
    title   = {Generating Physically Stable and Buildable LEGO Designs from Text},
    author  = {Pun, Ava and Deng, Kangle and Liu, Ruixuan and Ramanan, Deva and Liu, Changliu and Zhu, Jun - Yan},
    journal = {arXiv preprint arXiv:2505.05469},
    year    = {2025}
}

Model Card Contact

Ava Pun (apun@andrew.cmu.edu)

Framework versions

• PEFT 0.15.0