Content
summary Summary

NeRFshop lets you edit objects in NeRFs directly. The project shows what the future of creative work could look like.

Ad

Neural Radiance Fields (NeRFs) are a neural 3D representation of scenes or objects and have become a central technology for neural graphics in recent years. Tools such as Nvidia's InstantNGP, Nerfstudio, or applications such as Luma AI make it possible to learn 3D scenes from a simple video recording and even maintain the correct exposure.

NeRF cannot be edited until now

However, the complex representations in neural networks make it difficult to edit content directly in NeRFs. While there are projects such as RecolorNeRF that can be used to change the color palette in NeRFs, extensive changes to the shape of objects, for example, are not yet possible.

With NeRFshop, European researchers are now demonstrating a method for changing the position of objects within NeRFs or adjusting their shape. Users can mark an object with a simple brush within InstantNGP, and an algorithm converts the mark into a 3D region that can be adjusted. This region can then be transformed into a tetrahedron.

Ad
Ad

This resulting bounding tetrahedron cage can then be manipulated selectively: A cup changes its position on a table, an excavator turns its shovel, or a character's fingers clench into a fist.

NeRFshop to be further improved

The changes in the NeRF can then be learned by re-training the NeRF, which can then be processed in the usual pipelines.

The team's work aims to drive new trends in neural scene manipulation, and they plan to explore methods for correcting inconsistencies in lighting, shadows, and highlights after processing. Processing introduces artifacts and noticeable distortions, but NeRFshop already reduces some of these.

Further improvements are possible as the cage gets finer. However, "making such a method sufficiently fast to maintain interactivity is an open challenge," the team said.

More information and examples are available on the NeRFshop project page. Another approach to making NeRF technology more accessible is the open-source Nerfstudio, which provides a simplified end-to-end process for creating, training, and visualizing NeRFs.

Ad
Ad
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.
Recommendation
Support our independent, free-access reporting. Any contribution helps and secures our future. Support now:
Bank transfer
Summary
  • NeRFshop lets you move, rotate and deform NeRF objects.
  • It runs in Nvidia's InstantNGP, and the modified NeRFs can then be further processed through the usual pipelines.
  • The team plans further improvements to address existing artifacts and lighting issues.
Max is managing editor at THE DECODER. As a trained philosopher, he deals with consciousness, AI, and the question of whether machines can really think or just pretend to.
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.