VR is a highly immersive medium. No doubt about that. VR players don’t just see the world through a screen as is the case in normal video games, they are literally inside that world and can interact with its objects and environments in an intuitive way. Most of my friends who tried VR the first time where shocked by the experience. They told me that they completely lost track of time and that they felt as if they moved to another reality. Simply put, they were deeply immersed.
There are several factors that contribute to immersion in an experience. One that is very important, and at the same time is often overlooked, is music.
Composers have always thought that music has the ability to increase the level of immersion of players experiencing digital content, be it videos, video games or VR. For interactive content, composers like Guy Whitmore, who are at the forefront of music making in non-linear settings, know that adaptive music can make a big difference for immersion (check this post for an explanation of what we mean when we say adaptive music). The reasoning is quite simple. With adaptive music, no matter how the user behaves, the music is always in sync with the emotions portrayed in the visuals and in the storyline. Here’s an example. My jolly village gets attacked by dark knights. The music, being adaptive, dynamically shifts from happy to dramatic. The double bass kicks in and the chords get more aggressive. In other words, the audio elements of the experience reinforce the story told through the visuals. Composers suggest that this reinforced feedback between different elements of an interactive experience increases immersion. This is intuitive and sounds like a plausible hypothesis, but… it’s still a hypothesis. No one had yet tested it out in the real world — until now!
So far, correlation between immersion and adaptive music has never been tested scientifically. So we decided to run our own psychological experiment to shed some light on this. In particular, the goal of our study was to understand the impact of music on immersion and on the time spent in a VR experience. Our main hypotheses, based on composers’ (and our) intuition were:
- The presence of music increases the level of perceived immersion in a VR experience.
- Deep adaptive music increases the level of immersion more than linear music.
- The presence of music increases the time spent in a VR experience.
- Deep adaptive music increases the time session in VR more than linear music.
- Deep adaptive music fits a VR scene better than linear music.
There are a couple of concepts I need to clarify so that you get the complete picture of these hypotheses. What’s deep adaptive music? What’s linear music? Linear music is the music we listen to all the time, for example on Spotify. It never changes every time it is played. It’s a composition that you record and can loop indefinitely, and that always goes through the same beginning, middle section and ending. By contrast, deep adaptive music is a very dynamic type of music that is generated in realtime by an Artificial Intelligence and that adapts to the emotional setting of a scene and to user interaction on the fly. Deep adaptive music is infinite in the sense that it will always be slightly different for each player that goes through an interactive experience. That’s because each player will explore a VR environment, for example, in a unique way.
To test our hypotheses we built a simple VR scene. The scene consisted of a space station with 2 rooms connected by a corridor. The first room (blue room) was peaceful and calm, the second (red room) was more aggressive. We modulated the mood of the rooms (tender vs angry) by using different lighting and levels of activity in the same objects present in both rooms.
We set 3 experimental conditions. Participants could explore the VR scene with no music, linear music and deep adaptive music. The music used was in an ambient-like style with electronic instruments. Have a look at the table below for a comparison between the 3 conditions.
|No Music||Linear Music||Deep Adaptive Music|
|A simple VR space station scene||Same VR scene||Same VR Scene|
|Two rooms connected by a corridor||A fixed, looping soundtrack||Same sound design|
|No Interactive objects in the scene||Composed in realtime by Melodrive|
|Each room has its own mood|
|Music adapts to the mood in each room|
We got 46 participants who took part in the experiment. We didn’t tell them what the experiment was testing, so they weren’t biased. Participants experienced only one of the 3 conditions and were instructed to explore the scene for as long as they liked. We tracked session time as an overall measure of engagement. Once participants were done exploring the VR experience, they were asked to fill out a questionnaire on music and immersion.
The day after we conducted the experiment we started analysing the data we collected. We were amazed by what we saw! The initial hypotheses we came up with were found to be true and supported by the empirical results. As an aside, all the results I discuss next are statistically significant (p<0.05).
The effect of deep adaptive music on immersion was huge. Indeed, deep adaptive music provides a 30% increase in immersion when compared to no music and 25% when compared to linear music.
The effect of deep adaptive music on time session was even more impressive. With deep adaptive music there’s a 42% boost in time spent in the VR scene over no music and a 27% increase over linear music.
90% of people agreed that music was a very important component that helped them feel immersed. Data showed that deep adaptive music contributes significantly more than linear music to the feeling of immersion.
We also found that deep adaptive music increases the fit between music and the VR scene by 49% when compared against linear music.
The qualitative feedback we got from participants was also great. A few participants who explored the scene with no music were a bit bored and thought that music could make the experience more immersive:
The silence made the entire experience very bland… the lack of sound took me out of the immersion.
It was interesting, would have been better with music.
Participants thought that the music overall was good and that it fitted the visuals well. However, some participants with linear music thought the music wasn’t changing to fit the unfolding experience:
I was listening closely for subtle changes in the music to indicate progression in some way, but it felt as though my actions and progressions through the rooms had no impact. I still enjoyed the music and it fit the theme very well.
I really enjoyed the atmosphere of the environment and the music up until the red room. At that point, it felt like the tone of the music no longer matched that of my surroundings.
Participants who got deep adaptive music noted that the music changed according to the changes in the scene. This reinforced their emotional journey:
I liked the sound. It was building tension. As I was moving through the scene I could feel it became more intense. When moving away from the red room the tensions decreased again.
The music did a great job of blending transformations based on the scene, adding arrangement layers when less or more action was present.
What Have We Learned?
The psychological experiment we conducted finally provides empirical proof to the long-standing assumption that music has a huge impact on interactive media in terms of immersion and engagement. We now know that music plays a fundamental role not only in increasing the perceived level of immersion, but also in the time spent in a VR experience. In the light of the findings of this study, we can easily say that good music can increase immersion by one third and almost increase the time session by half. That’s a massive impact!
For me, the most striking element that arises from the experiment is not the huge impact of music as a whole. Being a composer and a gamer, I was already convinced that music, being an effective medium to convey subtle emotional cues, would have a major impact on engagement. What I’m really shocked about is that there’s a huge difference between the impact of linear and deep adaptive music in VR. Of course, I was expecting for deep adaptive music to have a stronger effect on immersion and time session. But I couldn’t imagine the difference could be so big. The results suggest that deep adaptive music is what really makes the difference for engagement.
You can tell this also from the qualitative feedback that we collected. Participants were particularly good at spotting mismatches between visuals and music, and appreciated it a lot when the music coordinated with the VR scene. This is the final take-away for me. Music is key for engagement in VR, but to be more specific, deep adaptive music is what can make the difference in such a highly non-linear medium.
Think of how overlooked music is in VR right now and, nonetheless, how immersive of an experience VR is. Now, imagine you add the right music, deep adaptive music. With a simple move you’ll get at least a one-third increase in immersion. That’s an amazing result!
The preliminary results of this study have already been published at an international workshop. Check out the conference proceedings (our paper is at page 8), if you’d like to get more details about the experiment.