Nvidia shows Instant NeRF: The AI technology renders a high-resolution 3D scene within seconds based on a few photos.
Neural Radiance Fields (NeRFs) are neural networks that can learn a 3D scene from 2D images and then reproduce it. The technology creates photorealistic scenes and is central to the use of AI in computer graphics and photography. Google, for example, recently demonstrated a type of Street View 3D that uses NeRFs to render entire blocks of streets.
One obstacle to the use of NeRF has been the long training and rendering times of neural networks. In recent years, however, researchers have found new ways to reduce the time required. Google researchers last year demonstrated real-time rendering of NeRFs. In early 2022, Nvidia unveiled Instant-NGP, a method that greatly speeds up AI training.
Higher rendering speed paves the way for NeRFs to get into real-world applications
Now, as part of this year's GTC, Nvidia is introducing Instant NeRF. The technology enables training and rendering of a high-resolution 3D scene in a matter of seconds. Compared to other NeRF methods, Instant NeRF is up to 1000 times faster in some cases, Nvidia says.
The higher speed is made possible by the reliance on a combination of the neural network with a learned hash table of voxel vertices presented in Nvidia's Instant-NGP.
The AI model needs only a few dozen photos from different angles for training, including information about camera angles, and then learns a 3D representation of the scene in a few seconds. The rendering of the scene itself takes only milliseconds.
NeRFs are the JPEG compression for 3D
“If traditional 3D representations like polygonal meshes are akin to vector images, NeRFs are like bitmap images: they densely capture the way light radiates from an object or within a scene,” says David Luebke, vice president for graphics research at NVIDIA.
“In that sense, Instant NeRF could be as important to 3D as digital cameras and JPEG compression have been to 2D photography — vastly increasing the speed, ease and reach of 3D capture and sharing.”
According to Nvidia, Instant NeRF could be used for avatars or scenes in virtual worlds, to capture video conference participants and their surroundings in 3D, or to reconstruct scenes for 3D digital maps. The latter could then be used for training robots and self-driving cars.
Nvidia sees further applications in architecture and entertainment, where NeRFs could be used to create digital representations of real-world environments that are then adapted and enhanced by developers.
