RecolorNeRF can change the color of objects or lighting in NeRFs. This enables easy image processing for photorealistic 3D scenes.
Neural Radiance Fields learn 3D representations of a scene, replacing older methods such as photogrammetry. Once learned, the scene can be rendered from different angles or exported as a 3D object.
Researchers at the University of Hong Kong and the University of Hong Kong (Shenzhen) now present RecolorNeRF, a method for selectively changing the colors in the neural network.
Basic Photoshop for NeRFs
According to the team, RecolorNeRF is the first approach that can directly modify photorealistic NeRFs in high quality. The researchers divide the scene into different layers on a learned color palette, which they can then modify. In this way, they can edit the NeRF directly and then render the scene as usual.
In the paper, the team shows that the method produces realistic results even in complex real-world scenes. For example, RecolorNeRF changes the color of a car or the green grass behind the car.
Semantic analysis for complex NeRF processing
RecolorNeRF provides an easy way to change colors in a NeRF. Since layers are only decomposed at the color level, however, the method modifies the color of all objects that have the same color. Currently, it is not possible to change only a single object.
Still, methods like RecolorNeRF show the potential of NeRFs in the computer graphics world. In the future, more complex NeRF processing will be possible.
The Chinese team is also planning improvements to RecolorNeRF: semantic segmentation could decompose the scene into multiple layers based on semantic information instead of just a color palette, allowing for targeted processing of individual objects in the future.
More examples and the code are available on the RecolorNeRF project page.