summary Summary

Nvidia's new GET3D AI model is designed to speed up the creation of 3D content. The model outputs textured 3D polygon meshes based on 2D photos that can be used seamlessly in standard graphics engines. 

GET3D stands for "Generate Explicit Textured 3D" and is a generative 3D model that can synthesize high-quality 3D polygon meshes with any topology. A single 2D image is sufficient as input for the generation process.

The generated polygon meshes consist of textured triangles - a standard format that allows seamless import into 3D programs, game engines, or movie renderers.

The 3D objects are fully editable after import and can be scaled, rotated, and lit, for example. With Nvidia's StyleGAN-Nada, developers can further change the shape or texture of the 3D model using only text commands and thus, for example, transform a conventional car into a police car.


3D model generation from synthetic 2D images

Nvidia's research team has developed a two-step generation process: The geometry branch generates the polygon mesh with any desired topology. The texture branch generates a texture field that can represent colors and, for example, specific materials at the surface points of the polygon mesh.

Finally, as with GA networks, discriminators evaluate the quality of the output based on synthetic photos of the 3D model and continuously optimize it to match the target image.

The training process of GET3D. | Image: Nvidia

GET3D was trained with about one million synthetic 2D images of 3D models from different angles. According to Nvidia, the training took about two days on Nvidia A100 GPUs.

GET3D accelerates the 3D content process

The 3D models that GET3D can generate depend on the training data: For example, if you train the system with synthetic car or animal images, it can generate 3D cars or animals. The larger and more diverse the training data set, the more detailed and diverse the 3D models generated, Nvidia says.

On a single off-the-shelf Nvidia GPU, the model can generate around 20 shapes per second after training, which combine to form a 3D model, according to Nvidia. The generation takes place locally on the user's computer and is thus independent of content restrictions, such as those known from cloud AI services.


"GET3D brings us a step closer to democratizing AI-powered 3D content creation," says Sanja Fidler, head of Nvidia's research lab in Toronto, where the tool was developed.

One limitation of GET3D, according to Nvidia's research team, is that training is currently only possible with 2D silhouettes of synthetic images from known camera positions. In future versions, advances in camera position estimation could form the basis for training with real images.

Currently, GET3D would also be trained only per category. A cross-category model could increase the variety of 3D models generated and improve the flexibility of the system.

As an open-source model, GET3D is available for free on Github.

Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.
Support our independent, free-access reporting. Any contribution helps and secures our future. Support now:
Bank transfer
  • Nvidia's 3D model GET3D can generate up to 20 3D shapes per second that combine to form textured 3D models based on 2D photos.
  • These 3D models comply with established graphics standards and can be refined and further processed in common 3D workflows.
  • GET3D is available as an open-source model for free on Github.
Online journalist Matthias is the co-founder and publisher of THE DECODER. He believes that artificial intelligence will fundamentally change the relationship between humans and computers.
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.