Microsoft's "PeopleLens" combines artificial intelligence with augmented reality: The system helps blind people interact better with their immediate social environment. Blind children in particular are expected to benefit.
Social participation can be a major challenge for blind and visually impaired people. Particularly affected are blind children who have difficulty making social contacts and friendships with their peers. Fundamental differences in the way sighted and blind children learn attention processes in infancy and early childhood are considered a possible cause.
For example, sighted children internalize a type of shared visual dialogue of attention, such as when the infant points to an animal and the parent says "cat." Children thus learn, for example, how to direct the attention of others.
Corresponding research on the formation of joint attention in blind children has been scarce. But for those affected, social interaction with sighted people is challenging.
PeopleLens works like a sound visor
Children born blind may therefore have difficulty directing their voice to conversational partners or maintaining a topic of conversation. Instead, they only talk about a single topic that interests them. As a result, affected children find it difficult to make friends. This is a burden for the children and their families.
PeopleLens is a Microsoft research project designed to help blind children interact more easily with their peers. The technology combines computer vision with Nreal Light AR glasses connected to a smartphone. The artificial intelligence system continuously locates, identifies and tracks people and their line of sight in the immediate environment of the AR glasses wearer.
Blind children receive the positioning information in the form of a spatial sound: a tone signals that their gaze has crossed that of a person up to ten meters away. If the person is within a radius of four meters, their name is also played - provided he or she has previously registered in the system via smartphone photo. Another audio system finds and centers a person's face.
PeopleLens to simplify social participation
The glasses also show those being tracked that they have been seen and can communicate back. This is to serve as a substitute for eye contact, which normally initiates interaction between sighted people.
Children participating in the research project additionally play a series of games with their peers that are designed to promote the development of spatial attention skills, thus supporting social interaction.
According to Microsoft, this is how blind children experience their agency in social interactions, such as when they realize they can start a conversation by themselves because they perceive a person first.
Microsoft, in collaboration with the University of Bristol, is currently recruiting blind children between the ages of five and eleven for a three-month study that will begin in September 2022. The study is intended to help improve the PeopleLens prototype and gain further practical experience.
