Google has detailed an experimental method of bringing facial expressions into virtual reality. It creates a more social experience by letting spectators see the facial expressions and gestures of people wearing a virtual reality headset.
The work builds on Google’s previous headset removal research announced back in February. The company is trying to address one of the most pressing issues with virtual reality headsets. Because they fully enclose your face, real world observers can’t see you or gauge your emotions.
The first version of the tech tracked eye gazes and blinks to work out where people in the scene were looking. It didn’t offer support for facial expressions though, leaving VR observers unable to engage with the social cues and gestures of the headset wearer.
The updated headset removal system uses deep learning to overcome this limitation. In a blog post late last week, Google explained how it can infer a subset of facial expressions from the data collected by the eye trackers. No additional hardware is required as the artificial intelligence dynamically detects the expressions.
Using eye trackers alone makes identification of facial features “a challenging task.” After training the system with data captured from 46 test participants, Google passed the resulting output through an advanced classification model. It then observed a success rate that’s good enough to make the technology feasible.
Google reiterated its previous calls for more research to be completed around VR headset removal. It noted there is considerable industry hype around social virtual reality experiences. These will be hindered if the headset wearer remains isolated from the people in the physical world around them.
“The ability to capture a user’s facial expressions using existing eye-tracking cameras enables a fully mobile solution to facial performance capture in VR, without additional external cameras,” said Google. “This technology extends beyond animating cartoon avatars; it could be used to provide a richer headset removal experience, enhancing communication and social interaction in VR by transmitting far more authentic and emotionally textured information.”
The technology addresses the restrictions around the physical design of headsets because of the opaque central display. It’s an issue the recently relaunched Google Glass avoids entirely by using a transparent head-mounted display form factor.
Google will be continuing to develop its technology as it tries to find a compelling alternative to the closed VR headset. The headset removal system is being demonstrated at Siggraph 2017 Emerging Technologies this week where attendees will be able to see the system in action.
This article was originally published by Digital Journal. Read the original article.