Content
summary Summary

A team of researchers presents the first high-quality, real-time rendering solution for radiation fields with training times in minutes and rendering at over 100 FPS.

Researchers from Inria, the Max Planck Institute for Informatics, and the Université Côte d'Azur have introduced a new approach for real-time, high-quality rendering of radiance fields acquired from real scenes. Their method achieves state-of-the-art visual quality while requiring only short training times, competitive with previous fast radiance field methods, and real-time rendering well above the gold standard of 30 fps.

Video: Kerbl, Kopanas et al.

The key is to represent the scene with a set of 3D Gaussians, essentially a flexible 3D shape that can also be rasterized very efficiently by projecting it into 2D. The method starts with a basic set of points obtained from a standard camera calibration technique called structure-from-motion (SfM). From these points, the method creates the 3D Gaussians. Each Gaussian has a central position, a matrix that specifies its shape and direction, and its level of transparency (opacity). Using the input images as a reference and other techniques, the method then fine-tunes the rendering by adjusting the Gaussians.

Ad
Ad

High-quality rendering with over 100 FPS

For fast rendering, the authors developed a GPU rasterizer that projects the 3D Gaussians to 2D. This tool organizes the Gaussians in a specific order and ensures that they blend correctly. It also does so quickly, while preserving the necessary details of each image.

The method achieves real-time rendering of 1080p novel views at over 100 FPS, with quality that matches or exceeds the best prior work, such as Mip-NeRF360 which needs up to 48 hours of training time. Optimization of 3D Gaussians takes only minutes, competitive with the previous fastest methods such as InstantNGP.

"First approach that truly real-time, high-quality radiance field rendering"

According to the team, the 3D Gaussians provide a compact, flexible scene representation that sets a new standard "We have presented the first approach that truly allows real-time, high-quality radiance field rendering, in a wide variety of scenes and capture styles, while requiring training times competitive with the fastest previous methods," they write.

Since more than 80 percent of the training time is spent on Python code, the team suspects further work, as porting the remaining optimization entirely to CUDA could provide significant further speedup for applications where performance is critical.

More information, examples and code are available on the project website.

Ad
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.
Ad
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.
Recommendation
Support our independent, free-access reporting. Any contribution helps and secures our future. Support now:
Bank transfer
Summary
  • Researchers from Inria, Max Planck Institute for Informatics, and UniversitĂ© CĂ´te d'Azur have developed a new method for real-time, high-quality rendering of radiance fields using 3D Gaussians, achieving state-of-the-art visual quality and fast rendering speeds.
  • The method starts with points obtained from camera calibration and creates 3D Gaussians, which are then fine-tuned and projected into 2D using a GPU rasterizer for fast rendering while maintaining image details.
  • This approach allows for real-time rendering of 1080p views at over 100 FPS with quality matching or exceeding prior work, and the team believes further optimization, such as porting to CUDA, could increase performance for applications where speed is critical.
Sources
Max is managing editor at THE DECODER. As a trained philosopher, he deals with consciousness, AI, and the question of whether machines can really think or just pretend to.
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.