PAC-NeRF demonstratse how NeRFs can learn the geometric structure and physical properties of objects from video.
Neural radiance fields (NeRFs) are a powerful AI-based rendering technology that can be used for video production, 3D reconstruction, and other tasks. They learn to represent and render the geometric structure and lighting of scenes and objects from video. But there are still limitations: the scenes are static and don't represent the physical properties of objects.
Researchers at UC Los Angeles, University of Maryland, MIT CSAIL, Columbia University, UMass Amherst, and the MIT-IBM Watson AI Lab are now taking NeRFs a step further: PAC-NeRF (Physics Augmented Continuum Neural Radiance Fields) learns both the geometric structure and physical properties of objects. This allows the network to represent dynamic scenes, and the learned physical properties could be used for other pipelines in the future.
Left video, center PAC-NeRF, right particle reconstruction. | Videos: Li et al.
PAC-NeRF follows conservation laws of continuum mechanics
The researchers are developing an architecture for PAC-NeRF that forces the network to follow the conservation laws of continuum mechanics during training. In this way, the team ensures that NeRF models only physically plausible states. The representation used "seamlessly blends efficient neural rendering with the material point method (MPM) for robust differentiable physics simulation."
PAC-NeRF can represent objects with properties of Newtonian and non-Newtonian fluids, as well as elastic materials, plasticine, or sand. Training a mesh currently takes 1.5 hours on an Nvidia 3090 GPU, rendering one image takes about a second. The team sees its own method as significantly superior to older approaches that forgo learning geometric structure or do not incorporate physical laws.
PAC-NeRF should be able to model more materials in the future
The team sees the current need for fixed camera angles with synchronized and calibrated cameras as a limitation. Also, the scene should have a simple and easily removable background. Most of the examples shown therefore use footage from a physical simulation, but there is also a real-world example.
Left video, center PAC-NeRF, right particle reconstruction. | Video: Li et al.
In the future, the team plans to extend the MPM framework used to train objects with other physical properties, such as cloth, stiff materials, or joint body simulations.
More examples are available on the PAC-NeRF project page.
