By Jen Brewster.
Virtual Reality is an evolving medium in today’s world, with artists and filmmakers alike learning the new visual language. However, with so many focused on the new visual storytelling, what place does audio take on that list of priorities?
Good sound, as with traditional formats, can add so much to a VR production, enhancing the immersive nature of the 360 screen tenfold.
We sat down with specialists Angelina Phengphong (VR Producer) and Stuart St. Vincent Welch (VR Sound Engineer) of Nylon Studios to find out why sound is just as important as images in the medium of virtual reality.
What do you think VR can offer audiences over traditional formats?
AP – What is unique to VR is that it offers a fully immersive experience. It can allow you to be somewhere else, and even be someone else. For the time you’re in the headset, your mind temporarily leaves the body to travel to another world. It’s pretty amazing. In these worlds, you also have some form of agency. Whether it’s choosing where to look or interacting with objects and people, you have a choice of how you want to experience the world around you; which is an important aspect to creating immersive environments. It’s currently the only format that can give you an experience that competes with reality. You can interact with something and that something, or someone, can react back. Of course, it’s not quite there yet, but most VR experiences these days immerse you into a world you cannot help but embrace fully, with no outside disturbances.
Why is sound so important in VR, surely it’s all about the images, right?
AP – Not anymore, and for the same reason that VR differentiates itself from any other media: immersion. You cannot be fully immersed, fully present and trick your senses into believing, if your aural experience doesn’t match your visual. When have you ever been in a room, a field, a car, anywhere in the real world and been in complete silence? Unless you’re in a soundproofed room, you’re never in complete silence. It could be the ticking of a clock, the muffled sound of the neighbours arguing two floors below you, or even the sound of your virtual-self breathing. Not only is it about the types of audio cues, but how they sound. A clap in an open field will resonant differently to a clap in a cathedral. It’s these minute details, that we often don’t take notice of, that can be the difference between a one-off novel experience and a truly immersive experience that leaves you wanting more.
What are the main audio tricks you can use to enhance a VR experience?
SSVW – The technology that produces spatial sound is highly advanced and consistently evolving to allow the user to hear objects all around you. You can make objects feel like they are passing right by your ear or long distances away. By using audio cues – a monster’s roar, a creaking door, a crow’s caw – directors now can tell stories and ‘direct’ the viewer’s gaze the way they want, rather than using crude and obvious pointers, either visual or aural, that can take you out of the experience.
There are also some great audio components that really heighten the realism of VR, such as audio backpacks and suits which let you feel the bass of objects in action like explosions, impacts, planes or cars. So it’s really the synergy of all the senses that create an immersive experience. We’ve taken our decades of experience in the sound & music craft to refine our VR audio process over the past year. We’re very proud of the work we’ve done and offer a premium service to our client…but we wouldn’t want to give away any of our “tricks” just yet!
How do you capture that kind of spatial audio if you’re shooting filmed 360 content on location?
SSVW – There are two options for this: the first is a quad mic system which captures 3D spatial audio which you can place in the VR world. It replicates the sound and movement exactly as you would hear on location in your headset. The issue with this is it won’t capture individual dialogue effectively at certain distances, and there can be unwanted noise produced from the camera/Drones. The second option we have found more effective, much like standard film location sound, is to mike up all the actors. We then place the audio in the location and follow the movement of characters and objects in post by placing the audio at the source’s position within a 360-degree space.
How do you create that kind of spatial audio if you’re creating a fully CG piece of VR content?
SSVW – The future of VR is being built in gaming engines, such as Unity and Unreal Engine. The level of interactivity and depth they offer create the most immersive VR experience to date. This is also true in the audio sense. It allows you to have interactive localized sounds that seamlessly replicate the sound of the real world. The way we go about doing this is supplying audio assets to the programmer, who attach the sounds to certain characters and objects. This is usually done before the CGI is created and requires a lot of preplanning so everyone is on the same page about what they want the objects and characters to sound like. Any scene can require up to and above 100 plus audio components running at the same time, which can become very labour and CPU heavy. We then remain closely involved through the production process to consult on the audio mix until the end. The interactive nature of VR, paired with convincing audio, is a crucial key to immersive VR.
Can you give any examples where you think spatial audio was key to a successful experience?
AP – There is a great range of examples where sound really takes the VR experience to another level. An obvious example is Notes on Blindness, which is a beautiful example of how VR not only lets you experience what it may be like to be another person, i.e. a blind person, but also utilises the platform and audio in a way that makes this experience only achievable in VR. I’ve found that horror experiences in VR demonstrate how sound design can really get inside your head. From the basic, yet effective, Face Your Fears on Oculus Gear to the recently released Wilson’s Heart – I’ve had to take the headset off for moments that, funnily enough, feel a little too real!