VideoRF paves the way for real-time 3D streaming on smartphones and VR headsets
Key Points
- Researchers present VideoRF, a new approach to render dynamic NeRF (Neural Radiation Field) scenes on various devices, including mobile platforms such as smartphones or VR headsets.
- VideoRF converts a learned NeRF into a simpler 2D image sequence that can be efficiently compressed, decoded, and rendered in real-time on mobile devices.
- Despite successes in creating 360° video sequences, complex multi-view acquisition systems and long training times are still required for longer sequences.
Researchers have introduced VideoRF, a new approach to streaming dynamic NeRF scenes on various devices, including mobile platforms.
Neural Radiation Fields (NeRFs) allow highly detailed 3D representations of scenes to be generated from a series of 2D images. Older methods have difficulty rendering moving scenes and are resource-intensive. However, there are now variants of NeRFs and related methods that are capable of displaying moving scenes and require significantly less time and computing power to train and render.
With VideoRF, a team from ShanghaiTech University, KU Leuven, and NeuDim demonstrates a method that enables real-time streaming of dynamic NeRFs to mobile devices such as smartphones or VR headsets.
VideoRF streams 360° video sequences of people in real-time
VideoRF converts a trained NeRF into a simpler 2D image sequence. This conversion is supported by special training to impose the temporal and spatial redundancy of the feature image stream. As a result, the images can be efficiently compressed by 2D video codecs and can take advantage of available hardware acceleration.
According to the researchers, the method enables real-time decoding and high-quality rendering on mobile devices at comparatively low and variable bit rates of around half a megabyte. The team also presents an interactive player for seamless online streaming and rendering of dynamic scenes.
In tests, the team shows that VideoRF can render immersive 360° video sequences of people. The ability to render dynamic neural radiation fields in real-time on mobile devices represents an important step in neural scene modeling, which is particularly important for VR/AR applications.
The team cites the complex multi-view capture systems required for the 360° video sequences as a limitation. In addition, long training times are still required, especially for longer sequences.
More information can be found on the VideoRF project page. The code should also be available there soon.
AI News Without the Hype – Curated by Humans
As a THE DECODER subscriber, you get ad-free reading, our weekly AI newsletter, the exclusive "AI Radar" Frontier Report 6× per year, access to comments, and our complete archive.
Subscribe now