Content
summary Summary

Instruct-NeRF2NeRF uses methods of generative AI models and can edit 3D scenes according to text input.

Ad

Earlier this year, researchers at the University of California Berkeley demonstrated InstructPix2Pix, a method that allows users to edit images in Stable Diffusion using text instructions. The method makes it possible to replace objects in images or change the style, for example.

Now some of the researchers have applied their method to editing NeRFs. Starting with a trained NeRF and the images used for training, Instruct-NeRF2NeRF can edit the training images one by one according to a text prompt and re-train the NeRF with these edited images.

Instruct-NeRF2NeRF supports simple objects and real scenes

The team shows how this method can be used to give a head a cowboy hat, turn it into a 3D oil painting, turn it into Batman, or turn it into Albert Einstein, for example.

Ad
Ad

Video: Haque et al.

Another example is changing the season, time of day, or weather of a nature shot; another is changing a person's clothes.

Video: Haque et al.

Video: Haque et al.

According to the team, the method is "able to edit large-scale, real-world scenes, and is able to accomplish more realistic, targeted edits than prior work."

Recommendation

Instruct-NeRF2NeRF updates the NeRF based on the iteratively edited images. This task is performed by InstructPix2Pix, which is conditioned by text input.

Instruct-NeRF2NeRF requires between 10 and 15 gigabytes of video memory

The team is releasing three different versions of Instruct-NeRF2NeRF that require between 15 and 10 gigabytes of RAM video memory.

The largest version gives the best results. The researchers cite the lack of ability to perform "large spatial manipulations" and the appearance of artifacts such as double faces as limitations.

More examples, code, and models are available on the Instruct-NeRF2NeRF project page. To learn how to create your own NeRFs, check out our no-code tutorial for Instant-NGP. Here we show you how to view NeRFs in VR.

Ad
Ad
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.
Support our independent, free-access reporting. Any contribution helps and secures our future. Support now:
Bank transfer
Summary
  • Instruct-NeRF2NeRF lets you edit NeRFs via text prompt. Behind this is the InstructPix2Pix team for image editing via Stable Diffusion.
  • The method can be used to edit faces, change the time of day, or change a person's clothing.
  • The team is releasing three versions that require between 10 and 15 gigabytes of video memory.
Sources
Max is managing editor at THE DECODER. As a trained philosopher, he deals with consciousness, AI, and the question of whether machines can really think or just pretend to.
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.