VR gaming with computer vision assisted audio coming
|Richard Harris in VR Thursday, March 21, 2019|
The world’s first computer vision assisted, spatial audio system could help virtual reality gaming ‘level up,’ claim experts from the University of Surrey’s Centre for Vision, Speech and Signal Processing (CVSSP).
In a paper to be presented at the IEEE VR Conference in Osaka Japan, researchers detail how they developed a computer-vision system that takes two images from off-the-shelf 360° cameras and uses AI to understand the acoustics of the user’s environment. This allows spatial audio to intelligently adapt to the user’s environment and produce a more immersive virtual or augmented reality experience.
The system uses a machine learning process called Convolutional Neural Network to analyze and extract a high-level understanding of the environment shape, type of objects and material acoustic properties, such as furniture, doors, windows, and carpet. The system takes this information to produce an acoustic model of the room which is used to adapt the spatial audio reproduction to the environment, therefore providing a realistic audio experience.
Dr. Hansung Kim, the lead author from CVSSP, said: “This is the fruit of long-term collaboration between vision and audio researchers in the S3A project at CVSSP. I believe this work has built a strong cross-disciplinary link between vision and audio research fields for complete VR/AR content production."
Professor Adrian Hilton, the co-author of the paper and Director of CVSSP, said: “This research advances the use of AI and computer vision to create a unique, immersive and compelling listening experience for users. This could be a game changer for VR and AR applications.
“We tend to focus on the creation of highly realistic visual content for VR but spatial audio is equally, if not, more important to engage participants and open immersive experiences to a mass audience.”
This research was conducted as part of S3A: Future Spatial Audio for Immersive Listener Experiences at Home – a flagship project funded by the EPSRC in collaboration with the BBC and Universities of Salford and Southampton.
Are you paying more taxes than you have to as a developer or freelancer? The IRS is certainly not going to tell you about a deduction you failed to take, and your accountant is not likely to take the time to ask you about every deduction you’re entitled to. As former IRS Commissioner Mark Everson admitted, “If you don’t claim it, you don’t get it.
Get hands-on experience in performing simple to complex mobile forensics techniques Retrieve and analyze data stored not only on mobile devices but also through the cloud and other connected mediums A practical guide to leveraging the power of mobile forensics on popular mobile platforms with lots of tips, tricks, and caveats.
The Chirp GPS app is a top-ranked location sharing app available for Apple and Android that is super easy to use, and most of all, it's reliable.
Write and run code every step of the way, using Android Studio to create apps that integrate with other apps, download and display pictures from the web, play sounds, and more. Each chapter and app has been designed and tested to provide the knowledge and experience you need to get started in Android development.