Content
summary Summary

PAC-NeRF demonstratse how NeRFs can learn the geometric structure and physical properties of objects from video.

Ad

Neural radiance fields (NeRFs) are a powerful AI-based rendering technology that can be used for video production, 3D reconstruction, and other tasks. They learn to represent and render the geometric structure and lighting of scenes and objects from video. But there are still limitations: the scenes are static and don't represent the physical properties of objects.

Researchers at UC Los Angeles, University of Maryland, MIT CSAIL, Columbia University, UMass Amherst, and the MIT-IBM Watson AI Lab are now taking NeRFs a step further: PAC-NeRF (Physics Augmented Continuum Neural Radiance Fields) learns both the geometric structure and physical properties of objects. This allows the network to represent dynamic scenes, and the learned physical properties could be used for other pipelines in the future.

Left video, center PAC-NeRF, right particle reconstruction. | Videos: Li et al.

Ad
Ad

PAC-NeRF follows conservation laws of continuum mechanics

The researchers are developing an architecture for PAC-NeRF that forces the network to follow the conservation laws of continuum mechanics during training. In this way, the team ensures that NeRF models only physically plausible states. The representation used "seamlessly blends efficient neural rendering with the material point method (MPM) for robust differentiable physics simulation."

PAC-NeRF can represent objects with properties of Newtonian and non-Newtonian fluids, as well as elastic materials, plasticine, or sand. Training a mesh currently takes 1.5 hours on an Nvidia 3090 GPU, rendering one image takes about a second. The team sees its own method as significantly superior to older approaches that forgo learning geometric structure or do not incorporate physical laws.

PAC-NeRF should be able to model more materials in the future

The team sees the current need for fixed camera angles with synchronized and calibrated cameras as a limitation. Also, the scene should have a simple and easily removable background. Most of the examples shown therefore use footage from a physical simulation, but there is also a real-world example.

Left video, center PAC-NeRF, right particle reconstruction. | Video: Li et al.

In the future, the team plans to extend the MPM framework used to train objects with other physical properties, such as cloth, stiff materials, or joint body simulations.

Recommendation

More examples are available on the PAC-NeRF project page.

Ad
Ad
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.
Support our independent, free-access reporting. Any contribution helps and secures our future. Support now:
Bank transfer
Summary
  • PAC-NeRF learns the geometric structure and physical properties of objects from video.
  • The team behind PAC-NeRF relies on physical priors in the network: the conservation laws of continuum mechanics.
  • PAC-NeRF can be used to learn dynamic NeRFs; next, the team wants to extend its framework to be able to learn more properties.
Sources
Max is managing editor at THE DECODER. As a trained philosopher, he deals with consciousness, AI, and the question of whether machines can really think or just pretend to.
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.