Content
summary Summary

Google's SMERF can stream 3D representations of entire homes to a smartphone in real-time.

Ad

Researchers from Google, Google DeepMind, and the University of Tübingen have unveiled SMERF, a new technology that enables real-time 3D rendering of large scenes, such as entire homes, on a variety of devices, including smartphones and laptops.

SMERF, which stands for Streamable Memory Efficient Radiance Fields, is a method based on NeRFs that relies on the more memory-efficient MERFs (Memory-Efficient Radiance Fields).

SMERF learns from Zip-NeRF

SMERF specializes in learning large 3D representations, such as entire homes of participating researchers. The team combines a hierarchical model partitioning scheme in which different parts of the space and the learned parameters are represented by different MERFs. This increases model capacity while limiting computational and memory requirements - such large 3D representations cannot be rendered in real-time with classical NeRFs.

Ad
Ad

To improve the rendering quality of SMERF, the researchers use a "teacher-student" distillation, in which an already trained, high-quality Zip-NeRF model (the "teacher") is used to train a new MERF model (the "student"). This approach allows the researchers to transfer the detail and image quality of the more powerful Zip-NeRF model to a more efficient and faster structure, which is particularly useful for applications on less powerful devices such as smartphones and laptops.

SMERF streams photorealistic 3D spaces in real-time via a web browser

Once trained, SMERF enables full six degrees of freedom navigation in a web browser and real-time rendering on popular smartphones and laptops.

The ability to render large 3D scenes in real-time is important for a variety of applications, including video games, virtual and augmented reality, and professional design and architecture applications. At Google, it could be used for Immersive View, for example.

However there are limitations: While the method offers excellent reconstruction quality and storage efficiency, it is associated with high storage costs, long loading times, and a high training effort. However, the work shows that NeRFs and similar radiation fields can still have advantages over 3D Gaussian splatting.

More information can be found on the SMERF project page. There are also demos showing real-time streaming of flats and other environments.

Ad
Ad
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.
Recommendation
Support our independent, free-access reporting. Any contribution helps and secures our future. Support now:
Bank transfer
Summary
  • Google, DeepMind, and the University of Tübingen announce SMERF, a technology for real-time 3D rendering of large scenes, such as homes, on smartphones and laptops.
  • SMERF (Streamable Memory Efficient Radiance Fields) is based on Memory-Efficient Radiance Fields (MERFs) and uses a hierarchical model partitioning scheme for memory-efficient rendering, as well as teacher-student distillation from a Zip-NeRF for better rendering quality.
  • Despite excellent reconstruction quality and memory efficiency, high storage costs, long loading times, and high training effort still exist.
Sources
Max is managing editor at THE DECODER. As a trained philosopher, he deals with consciousness, AI, and the question of whether machines can really think or just pretend to.
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.