How immersive audio will transform virtual experiences
Sony’s 360 Reality Audio turns music into a 3D world – and all you need to experience it is a pair of ordinary headphones. Here’s how Kojey Radical has been using it to create new forms of art.
The British rapper Kojey Radical is strolling through a vast museum filled with imposing classical statues, singing his hit single “Good”. As he begins to dance, something unexpected happens: the statues start to crumble and then explode violently, taking the museum with them. In the burning ashes, new statues appear depicting black figures towering high above the musician.
Of course, it isn’t real – it’s a music video made with CGI. But if you listen to it with headphones, you become part of this make-believe realm. That’s because it was mixed using Sony’s 360 Reality Audio.
This new format allows individual sounds to be precisely “positioned” anywhere around the head of the listener. Artists can choose to give their audience the experience of being on stage, say – spatially locating the drums, vocals and guitars exactly as they would be in the performance. Or, as in Kojey Radical’s project, they can create a whole virtual world – a vortex of stones swirling around your head, the sound of a hi-hat moving over one’s head from right to left – in a way that makes stereo feel decidedly basic. The most counterintuitive thing about it? You don’t need any special kit to make it work. An ordinary pair of headphones is all it takes to enjoy this immersive audio experience.
Kojey Radical is just one of a number of musicians who are experimenting with 360 Reality Audio to create new forms of artistry. His “Good” video is part of Unknown Realities, a campaign dedicated to showcasing the possibilities that arise when the format is teamed with CGI. “Unknown Realities has allowed me not only to tell a story, but also to create a world in which it can come to life,” he says of the piece he has created for his single “Good”. “I really wanted the energy and the message to be felt, and I believe 360 Reality Audio allowed this to happen.”
The video is available alongside a host of other 360 Reality Audio content at music.com/360RA – but listeners can also enjoy the format across a variety of streaming services: Deezer, TIDAL and Nugs.net. Those who register with Music.com will get three months of free access to either Deezer and TIDAL’s high-quality “HiFi” audio services.
The format represents an evolution in the world of spatial audio. The basic idea has been around for decades – if we can emulate how sounds from different positions arrive at our eardrums, we can go to create the illusion of being inside a 3D space. That’s because a noise to the left, for instance, will reach the left ear fractionally before the right ear. Our brain uses the time difference between those two events to subconsciously extrapolate the exact location. This is the principle behind binaural audio recordings such as Lou Reed’s 1978 album Street Hassle.
To achieve the effect, sound engineers often use a dummy head with a microphone in each ear. Naturally, these pick up the sounds of the room just as real ears would. When each channel’s recording is simultaneously played back through the respective speakers on a pair of headphones, the listener’s mind recreates the soundstage.
The problem is that everyone’s head and ears are shaped slightly differently. In practice, our brains use the precise contours of our ears, and the way in which this affects the sounds arriving from different directions, in order to pinpoint the origin of noises even more precisely. A dummy head (or a digital model of one) is intended to represent an average head and ear shape.
For the highest end experience, Sony offers a purpose-built app called Sony | Headphones Connect that works with Sony’s selected headphones. The user submits a photograph of their ears for processing by the app’s algorithm. This lets it adjust incoming sounds to simulate how they would register had they actually arrived from the top left, say, or the back right. It makes the whole effect incredibly persuasive.
What’s more, 360 Reality Audio doesn’t depend on physically recording sounds with a dummy head. Musicians can use software which visualises the “sphere” of sound around the listener. The artist can drag and drop elements such as drums or vocals into precise positions within this space. That means that everything from old jazz recordings to classic rock can be remastered for the format – and indeed the kinds of CGI visuals used in Kojey Radical’s Unknown Realities project can have their sound effects mapped into the experience.
Virtual worlds are becoming increasingly important to the wider culture. Musicians are holding concerts in Fortnite; faster mobile internet is making the virtual portable; and in recent years, more money than ever has poured into VR. As users come to expect greater immersion in how they consume film, games and music, “headphone virtualisation” formats such as 360 Reality Audio are only going to become more popular and more sophisticated. The question is: where does it go from here?