Content
summary Summary

In the project "Seeing the World through Your Eyes," researchers at the University of Maryland, College Park, show that the reflections of the human eye can be used to reconstruct 3D scenes. This, they say, is an "underappreciated source of information about what the world around us looks like".

Ad

The paper uses a NeRF-based method to reconstruct a 3D scene from a reflection in the eye. This sounds straightforward, but is actually a complicated process that involves several factors, including accurately determining the direction of gaze and distinguishing reflections from patterns in the iris, the researchers say.

Links sieht man eine Iris mit Spiegelung, rechts davon eine 3D-Rekonstruktion dieser Spiegelung.
In the "Seeing the World through Your Eyes" project, researchers show that the reflections of the human eye can be used to reconstruct 3D scenes. | Image: Alzayer, Zhang et al.

Calculating the outside world through the cornea

To calculate the outside world, the team uses the geometry of the cornea, which is fairly uniform in healthy adults. Based on the size of the cornea in an image, they can estimate the position and orientation of the eye.

An important aspect is optimizing the detection of the cornea's position, which is used to refine the initial position estimate for each image. This technique has proven to be critical to the robustness of the method.

Ad
Ad
The structure of the reconstruction method. | Image: Alzayer, Zhang et al.

To separate the reflection of the outside world from the iris pattern, the team used a modified training framework from Nerfstudio. They had NeRF learn both the reflected 3D scene and the iris pattern, and simultaneously trained the system to separate the two elements.

For iris detection, the system was given the general radially symmetric shape as a default and the assumption that the reflection changes from different perspectives but the texture of the iris remains the same.

Reconstructing a 3D scene from the reflection in the eye of a portrait photo. | Video: Alzayer, Zhang et al.

Bild im Fotolabor, Mann schaut in Kamera, auf dem Tisch vor ihm liegen zwei Plüschtiere, die Lampen leuchten die Szene hell aus.
The photo setup under laboratory conditions. | Image: Alzayer, Zhang et al.

Reconstruction of a 3D scene from the reflection in the eye of a portrait photograph. | Video: Alzayer, Zhang et al.

3D nerf with only one camera

While typical NeRF reconstructions require multiple camera views of the object being reconstructed, the researchers simply moved the person through the camera's field of view. The reflections in the eye, which change with even minimal movement, provided the different perspectives needed to reconstruct the 3D scene, even though only one camera saw the person.

Recommendation

To validate their method, the team used synthetic eye images rendered with Blender and real photographs of a person moving in the camera's field of view. Despite challenges such as inaccuracies in locating the cornea and estimating its geometry, and the inherently low resolution of the images, their method showed promise, they say. In tests with synthetic eye models, the team was able to perform full-scene reconstructions using only eye reflections.

Highly detailed reconstruction of a 3D scene using a synthetic eye model under optimal conditions. | Video: Alzayer, Zhang et al.

However, the tests were conducted only in the laboratory; in reality, many other factors are at play, and the assumptions made in the paper, such as about the texture of the iris, may be too simplistic. Brighter iris textures or situations involving strong eye rotation could pose additional challenges, the researchers write.

The team hopes the work will stimulate further research into how unexpected, random visual signals can be used to reveal information about the surrounding world.

Ad
Ad
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.
Support our independent, free-access reporting. Any contribution helps and secures our future. Support now:
Bank transfer
Summary
  • Researchers at the University of Maryland have developed a NeRF-based method to reconstruct 3D scenes from reflections in the human eye. They believe this is an underappreciated source of information about the world around us.
  • The method uses the uniform geometry of the cornea in healthy adults to estimate the position and orientation of the eye. An important aspect of the work is the development of a corneal position optimization technique that helps improve the robustness of the method.
  • Tests have been performed with both synthetic eye images and real photographs, but only under laboratory conditions. Despite certain challenges, such as inaccuracies in the localization of the cornea and the low resolution of the images, the method is considered promising.
Online journalist Matthias is the co-founder and publisher of THE DECODER. He believes that artificial intelligence will fundamentally change the relationship between humans and computers.
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.