NeRFshop lets you edit objects in NeRFs directly. The project shows what the future of creative work could look like.
Neural Radiance Fields (NeRFs) are a neural 3D representation of scenes or objects and have become a central technology for neural graphics in recent years. Tools such as Nvidia's InstantNGP, Nerfstudio, or applications such as Luma AI make it possible to learn 3D scenes from a simple video recording and even maintain the correct exposure.
NeRF cannot be edited until now
However, the complex representations in neural networks make it difficult to edit content directly in NeRFs. While there are projects such as RecolorNeRF that can be used to change the color palette in NeRFs, extensive changes to the shape of objects, for example, are not yet possible.
With NeRFshop, European researchers are now demonstrating a method for changing the position of objects within NeRFs or adjusting their shape. Users can mark an object with a simple brush within InstantNGP, and an algorithm converts the mark into a 3D region that can be adjusted. This region can then be transformed into a tetrahedron.
This resulting bounding tetrahedron cage can then be manipulated selectively: A cup changes its position on a table, an excavator turns its shovel, or a character's fingers clench into a fist.
NeRFshop to be further improved
The changes in the NeRF can then be learned by re-training the NeRF, which can then be processed in the usual pipelines.
The team's work aims to drive new trends in neural scene manipulation, and they plan to explore methods for correcting inconsistencies in lighting, shadows, and highlights after processing. Processing introduces artifacts and noticeable distortions, but NeRFshop already reduces some of these.
Further improvements are possible as the cage gets finer. However, "making such a method sufficiently fast to maintain interactivity is an open challenge," the team said.
More information and examples are available on the NeRFshop project page. Another approach to making NeRF technology more accessible is the open-source Nerfstudio, which provides a simplified end-to-end process for creating, training, and visualizing NeRFs.
