Content
summary Summary

DiffusionLight is a method that estimates lighting in an image using a generated chrome ball.

Ad

The researchers have developed a simple but effective technique for estimating illumination in a single input image. They use diffusion models trained on billions of standard images to render a chrome ball in the input image and use it as a light probe.

This method has a variety of applications, including realistic insertion of virtual objects into images, enhancements in AR and VR, realistic visualizations in architecture and interior design, more realistic scenes in computer games, and more accurate planning in photography and film.

DiffusionLight uses Stable Diffusion XL

Current lighting estimation techniques are based on HDR panoramic datasets that are used to train neural networks. However, these approaches often struggle with real-world images due to the limited variety and size of available datasets.

Ad
Ad

In contrast, the technique called "DiffusionLight" uses the Stable Diffusion XL diffusion model, pre-trained with billions of images, to render a chrome ball in the input image. The method assumes that the AI models have learned about HDR and the wide range of brightness indirectly from examples of underexposed and overexposed images in their training sets.

 

DiffusionLight provides the necessary information to adapt the lighting of placed 3D objects to that of the environment. | Video: Phongthawee, Chinchuthakun et al.

However, without further intervention, SDXL generates incorrect or inconsistent objects and cannot easily generate images in HDR format. The researchers therefore used a technique called "iterative inpainting" to find an initial diffusion noise map, which is then used to generate chrome balls of consistent quality.

To create HDR chromospheres, the researchers also generate and combine multiple LDR chrome balls with different exposure values, which are used to further refine SDXL with LoRA.

Recommendation

After training, DiffusionLight provides convincing light estimates in different settings and can process very different scenes.

More information and examples can be found on the DiffusionLight project page.

Ad
Ad
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.
Support our independent, free-access reporting. Any contribution helps and secures our future. Support now:
Bank transfer
Summary
  • Researchers develop DiffusionLight, a method for estimating illumination in images using a generated chrome ball as a light probe. This technique has applications in AR, VR, architectural visualization, and computer games.
  • DiffusionLight uses the Stable Diffusion XL model trained on billions of images to provide more realistic lighting estimates than traditional HDR panoramic datasets.
  • The researchers use "iterative inpainting" and a special LoRA to generate consistent chrome balls and HDR images, allowing DiffusionLight to deliver convincing lighting estimates in different environments.
Sources
Max is managing editor at THE DECODER. As a trained philosopher, he deals with consciousness, AI, and the question of whether machines can really think or just pretend to.
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.