Researchers have introduced VideoRF, a new approach to streaming dynamic NeRF scenes on various devices, including mobile platforms.
Neural Radiation Fields (NeRFs) allow highly detailed 3D representations of scenes to be generated from a series of 2D images. Older methods have difficulty rendering moving scenes and are resource-intensive. However, there are now variants of NeRFs and related methods that are capable of displaying moving scenes and require significantly less time and computing power to train and render.
With VideoRF, a team from ShanghaiTech University, KU Leuven, and NeuDim demonstrates a method that enables real-time streaming of dynamic NeRFs to mobile devices such as smartphones or VR headsets.
VideoRF streams 360° video sequences of people in real-time
VideoRF converts a trained NeRF into a simpler 2D image sequence. This conversion is supported by special training to impose the temporal and spatial redundancy of the feature image stream. As a result, the images can be efficiently compressed by 2D video codecs and can take advantage of available hardware acceleration.
According to the researchers, the method enables real-time decoding and high-quality rendering on mobile devices at comparatively low and variable bit rates of around half a megabyte. The team also presents an interactive player for seamless online streaming and rendering of dynamic scenes.
In tests, the team shows that VideoRF can render immersive 360° video sequences of people. The ability to render dynamic neural radiation fields in real-time on mobile devices represents an important step in neural scene modeling, which is particularly important for VR/AR applications.
The team cites the complex multi-view capture systems required for the 360° video sequences as a limitation. In addition, long training times are still required, especially for longer sequences.
More information can be found on the VideoRF project page. The code should also be available there soon.