A team at Carnegie Mellon University has introduced LegoGPT, an AI system that generates Lego models from text descriptions. The models aren’t just visually accurate—they can actually be built, either by people or by robots.
LegoGPT is powered by an autoregressive language model that predicts the next Lego brick in a build, similar to how text models predict the next word in a sentence. The researchers trained their system on a custom dataset called StableText2Lego, featuring over 47,000 stable Lego structures. The descriptions paired with each build were generated using GPT-4o based on 3D renderings.
Video: Pun, Deng, Liu et al.
Physics-based checks keep builds standing
As it generates each model, LegoGPT checks every brick placement for collisions, invalid positions, and physical instability. If the system detects a problem, it rolls back to the last stable state—a process the authors call "physics-aware rollback." Rejection sampling is also used to filter out invalid bricks before they’re added.
The stability analysis relies on a physics model that considers all forces acting on each brick, including compression, tension, and shear. Only designs that reach static equilibrium are accepted.
When tested against other 3D generation systems like LLaMA-Mesh, XCube, and Hunyuan3D-2, LegoGPT produced a much higher rate of stable, buildable models. While other methods often create unstable or impossible structures, LegoGPT achieves 98.8 percent stability across its results.
Real lego colors and AI textures
Lego models wouldn’t be much fun without color. LegoGPT can add both colors and textures to its creations. For example, a sofa described with "Japanese Shibori fabric" gets a structured indigo surface, while a "cyberpunk flame paint job" glows in neon violet. Texturing is handled by FlashTex, a fast text-to-mesh coating method.
In addition to UV textures, LegoGPT supports solid coloring using official Lego colors—so designs can work both digitally and in real-world builds.
More details and sample builds are available on the LegoGPT project page. There’s also a demo on HuggingFace.
