VR gaming with computer vision assisted audio coming
Thursday, March 21, 2019
The world’s first computer vision assisted, spatial audio system could help virtual reality gaming ‘level up,’ claim experts from the University of Surrey’s Centre for Vision, Speech and Signal Processing (CVSSP).
In a paper to be presented at the IEEE VR Conference in Osaka Japan, researchers detail how they developed a computer-vision system that takes two images from off-the-shelf 360° cameras and uses AI to understand the acoustics of the user’s environment. This allows spatial audio to intelligently adapt to the user’s environment and produce a more immersive virtual or augmented reality experience.
The system uses a machine learning process called Convolutional Neural Network to analyze and extract a high-level understanding of the environment shape, type of objects and material acoustic properties, such as furniture, doors, windows, and carpet. The system takes this information to produce an acoustic model of the room which is used to adapt the spatial audio reproduction to the environment, therefore providing a realistic audio experience.
Dr. Hansung Kim, the lead author from CVSSP, said: “This is the fruit of long-term collaboration between vision and audio researchers in the S3A project at CVSSP. I believe this work has built a strong cross-disciplinary link between vision and audio research fields for complete VR/AR content production."
Professor Adrian Hilton, the co-author of the paper and Director of CVSSP, said: “This research advances the use of AI and computer vision to create a unique, immersive and compelling listening experience for users. This could be a game changer for VR and AR applications.
“We tend to focus on the creation of highly realistic visual content for VR but spatial audio is equally, if not, more important to engage participants and open immersive experiences to a mass audience.”
This research was conducted as part of S3A: Future Spatial Audio for Immersive Listener Experiences at Home – a flagship project funded by the EPSRC in collaboration with the BBC and Universities of Salford and Southampton.
Become a subscriber of App Developer Magazine for just $5.99 a month and take advantage of all these perks.
MEMBERS GET ACCESS TO
- - Exclusive content from leaders in the industry
- - Q&A articles from industry leaders
- - Tips and tricks from the most successful developers weekly
- - Monthly issues, including all 90+ back-issues since 2012
- - Event discounts and early-bird signups
- - Gain insight from top achievers in the app store
- - Learn what tools to use, what SDK's to use, and more