Content
summary Summary

Advances in 3D reconstruction are enabling high-quality text-to-3D modeling, creating a 3D model from text in less than half an hour.

Ad

In August, a team presented 3D Gaussian Splatting, a new AI method that enables high-quality, real-time rendering of radiance fields. Using just a few dozen photos, it can render photorealistic 3D scenes in real time after a few minutes of training.

Since then, this method has attracted some attention in the field of 3D reconstruction and is accessible to everyone through applications such as Polycam.

Text-to-3D using diffusion and 3D Gaussian splatting

There is now some initial work using 3D Gaussian splatting to generate 3D models from text descriptions. They essentially follow known architectures, such as using a 3D diffusion model like OpenAI's Point-E to generate a point cloud from which to initialize the 3D Gaussians. Their geometry and appearance are then further refined using 2D diffusion models for image generation.

Ad
Ad

Source: Chen at al.

Similar approaches are used by DreamFusion or Magic3D. The 3D Gaussian methods such as GSGen3D or GaussianDreamer achieve similar or better results in much less time: while the old methods take several hours for an object, the new methods generate 3D objects in less than an hour.

GSGen3D and GaussianDreamer are just the beginning

Further improvements in quality and speed are expected. The ability of current models to handle complex prompts is still limited, with one of the authors citing the low text comprehension of Point-E and CLIP in Stable Diffusion as a reason. Better variants of these models could enable more complex 3D models with more precise prompting, similar to what OpenAI's DALL-E 3 demonstrates.

More examples, the code, and more information can be found on the GSGen3D and GaussianDreamer project pages.

Ad
Ad
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.
Support our independent, free-access reporting. Any contribution helps and secures our future. Support now:
Bank transfer
Summary
  • 3D Gaussian Splatting provides high-quality real-time rendering of radiance fields and is now making progress in generating 3D models from text descriptions.
  • Models such as DreamFusion and Magic3D have taken similar approaches, but the new methods such as GSGen3D and GaussianDreamer are faster and produce similar or better results.
  • Further improvements in speed and quality can be expected, as better variants of these models could enable more complex 3D models with more precise prompting, as demonstrated by OpenAI's DALL-E 3.
Max is managing editor at THE DECODER. As a trained philosopher, he deals with consciousness, AI, and the question of whether machines can really think or just pretend to.
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.