Google's SMERF can stream 3D representations of entire homes to a smartphone in real-time.
Researchers from Google, Google DeepMind, and the University of Tübingen have unveiled SMERF, a new technology that enables real-time 3D rendering of large scenes, such as entire homes, on a variety of devices, including smartphones and laptops.
SMERF, which stands for Streamable Memory Efficient Radiance Fields, is a method based on NeRFs that relies on the more memory-efficient MERFs (Memory-Efficient Radiance Fields).
SMERF learns from Zip-NeRF
SMERF specializes in learning large 3D representations, such as entire homes of participating researchers. The team combines a hierarchical model partitioning scheme in which different parts of the space and the learned parameters are represented by different MERFs. This increases model capacity while limiting computational and memory requirements - such large 3D representations cannot be rendered in real-time with classical NeRFs.
To improve the rendering quality of SMERF, the researchers use a "teacher-student" distillation, in which an already trained, high-quality Zip-NeRF model (the "teacher") is used to train a new MERF model (the "student"). This approach allows the researchers to transfer the detail and image quality of the more powerful Zip-NeRF model to a more efficient and faster structure, which is particularly useful for applications on less powerful devices such as smartphones and laptops.
SMERF streams photorealistic 3D spaces in real-time via a web browser
Once trained, SMERF enables full six degrees of freedom navigation in a web browser and real-time rendering on popular smartphones and laptops.
The ability to render large 3D scenes in real-time is important for a variety of applications, including video games, virtual and augmented reality, and professional design and architecture applications. At Google, it could be used for Immersive View, for example.
However there are limitations: While the method offers excellent reconstruction quality and storage efficiency, it is associated with high storage costs, long loading times, and a high training effort. However, the work shows that NeRFs and similar radiation fields can still have advantages over 3D Gaussian splatting.
More information can be found on the SMERF project page. There are also demos showing real-time streaming of flats and other environments.