A team of researchers presents the first high-quality, real-time rendering solution for radiation fields with training times in minutes and rendering at over 100 FPS.
Researchers from Inria, the Max Planck Institute for Informatics, and the Université Côte d'Azur have introduced a new approach for real-time, high-quality rendering of radiance fields acquired from real scenes. Their method achieves state-of-the-art visual quality while requiring only short training times, competitive with previous fast radiance field methods, and real-time rendering well above the gold standard of 30 fps.
Video: Kerbl, Kopanas et al.
The key is to represent the scene with a set of 3D Gaussians, essentially a flexible 3D shape that can also be rasterized very efficiently by projecting it into 2D. The method starts with a basic set of points obtained from a standard camera calibration technique called structure-from-motion (SfM). From these points, the method creates the 3D Gaussians. Each Gaussian has a central position, a matrix that specifies its shape and direction, and its level of transparency (opacity). Using the input images as a reference and other techniques, the method then fine-tunes the rendering by adjusting the Gaussians.
High-quality rendering with over 100 FPS
For fast rendering, the authors developed a GPU rasterizer that projects the 3D Gaussians to 2D. This tool organizes the Gaussians in a specific order and ensures that they blend correctly. It also does so quickly, while preserving the necessary details of each image.
The method achieves real-time rendering of 1080p novel views at over 100 FPS, with quality that matches or exceeds the best prior work, such as Mip-NeRF360 which needs up to 48 hours of training time. Optimization of 3D Gaussians takes only minutes, competitive with the previous fastest methods such as InstantNGP.
"First approach that truly real-time, high-quality radiance field rendering"
According to the team, the 3D Gaussians provide a compact, flexible scene representation that sets a new standard "We have presented the first approach that truly allows real-time, high-quality radiance field rendering, in a wide variety of scenes and capture styles, while requiring training times competitive with the fastest previous methods," they write.
Since more than 80 percent of the training time is spent on Python code, the team suspects further work, as porting the remaining optimization entirely to CUDA could provide significant further speedup for applications where performance is critical.
More information, examples and code are available on the project website.