A couple of weeks ago, on a beautiful Berlin summer’s day, the Melo-team set up shop for what was to turn out to be perhaps the bestest hack weekend ever: #EmoJam2. With a dollop of luck that we didn’t include in our plans, we successfully brought together VR gurus, music tech experts, designers, musicians, and emotional beings in one space to join forces for a chillingly good dash of problem-solving and artistic collaboration. The inter-field love was blossoming; all attendees left their egos, but not their pride, at the door. One of our attendees, journalist Topper Sherwood, gauged the atmosphere perfectly, if you’ll permit me to paraphrase: the hack permitted a bridge between disciplines, giving opportunity for people to engage in each other’s specialities; allowing artist and scientist to join together in sweet harmony.
Indeed, we’ve seen some great things during this last weekend, from industry-insider talks, to the building of friendships, to state-of-the-art hacking by amateurs and professionals. In short, it’s been awesome. Let’s fill you in with what went down.
Firstly, following #EmoJam1, #EmoJam2 had the most amazing lineup of speakers, informing us on a range of subjects.
Kim Baumann Larsen
The first speaker, Kim Baumann Larsen, looked at psychology, emotion, and storytelling in VR spaces, and how various VR environments influence the mind; particularly how virtual spaces change emotions and alter states of consciousness. For example, he noted that big open spaces, such as churches, foster abstract thought. Many other spaces also have specific effects on brains too. According to Kim, the most inspiring view for observers is when you look out onto a huge, vast vista with borders, such as what you’d get from looking out of a cave onto a deep valley below. The emotive idea behind this is that one gets the feeling that they are situated in the cave in relative safety, yet they can view the boundless and sublime beauty of the expanse, creating a wonderful confliction. This was a really inspiring talk — thank you Kim!
Alexander Grobe gave fascinating insight after fascinating insight discussing measuring and influencing mental states. The use of technology to influence and measure emotions was the main thread. He described how he made a study about the influence of bass tones on the electrical activity in brains in VR experiences. The experiment used a Subpac personal subwoofer that sent out low frequencies perceived by the user via their chest. These can then be measured with EEG to show the effect on the user’s brain waves. Lo(w) and behold, the bass had a marked effect on experience. Alexander also gave lots of technical information on the gear he uses, explaining how to get greater immersion and engagement through various VR hacks. The super-immersive Subpac is an EDM producer’s dream, allowing that deep bass nightclub feel at your nipple-tips!. Alexander was the super-personality at the hack, abounding energy and good vibes. He generously donated his stuff for others to use too. The long and short is, if you don’t already get it, this guy is probably the coolest guy you’re ever gonna meet. Ever.
Linsey Raymaekers, who specialises in human-computer interaction, talked about games going beyond mere entertainment, extending to life-building experiences and emotional development. Game development doesn’t have to be about gaming, but can involve activity more serious and thought-provoking; it can be about interest, learning, and moral journeys. She’s also taken with the idea that games can be a means for self expression. And just when you thought that the concept of game wasn’t already complicated enough.
Linsey furnished the extended concept of game with pertinent examples of small-scaled indie games created by herself and others. One game was developed by a couple struggling to raise a child with cancer. Even though the couple hadn’t been developers previously, and had little experience in designing games, as a coping strategy, they invented the game That Dragon, Cancer. The project gave them the therapeutic needs they required, being almost a type of cathartic release. An important underlying point is that non-developers in the traditional sense can now become developers. Linsey herself has made several games, and is working with a company called Honig Studios. She has authored a game called Compass, which she demonstrated. The object of the game is to overcome one’s fears. It’s an experimental, personal game that’s an enabler of thought-provoking experiences — and you can also have fun on the side too, but yet it’s so much more than that.
Billy Mello, composer and sound designer, gave an interesting talk on how to guide gamers to an intended emotional state, showing how emotion in scenes can be changed by the hidden hand of music. The main focus was how music adds to (and takes away from) the emotion of a video, game, or interactive experience. He argued that the composer is in a unique position, because he can alter the received emotion through his own slant on the overall product. He had a great way to illustrate the point. Videos of games were displayed to us, but different, often contradictory types of music were played over the top. For instance, he first showed Thatgamecompany’s ‘Flower’. This is a colourful game where you float around as a flower petal breathing life into the land. The original music is very floaty, ambient and delicate, with wind and sporadic (Sporish) music playing as accompaniment. However, Billy put the visuals in black and white, and played dark suspenseful music with it — horror music. This changed the mood from its original feeling — bright, colourful and uplifting — to a more ominous and foreboding dread. This presentation really hit home, giving us food for thought about the power of music.
Byrke Lou gave an incredible talk about music and emotion in VR spaces. She emphasised the importance of appropriate music in abstract, fantastical and invented VR contexts. Traditional music can’t work when you can have a VR space that is completely fantastical, such as an invented and surreal abstract space. This insight might stem from Byrke’s own background in physics, which drives her to question the various potentials and possible unrealities of VR experiences. She profiled one of her student’s work, which is a dreamy, mind-bending VR space. Indeed, traditional music over the top would have been an imposition, and it would have taken away from the experience. The actual music was eerie and emergent, and suited the mood much better. Great talk!
Finally, our very own Valerio Velardo, polymath, supreme leader, rhetorical wizard, and fast thinker-on-his-feet, whetted our appetites by offering up free usage of the Melodrive SDK for the hackers to play with. If you don’t know what Melodrive is all about, I could sum it up by saying that we’re putting the deep into deep adaptive music (DAM), although Valerio summed it up just as nicely using different words. I could also sum it up by saying that Melodrive is an AI music machine that create AI adaptive music in realtime. Which is what Valerio also said. Valerio then dared the hack-folk to come up with something cool that they can achieve in their teams through the use of the Melodrive engine for the weekend, gently taunting the delegates with a ‘now go hack it and do something better!’
As if invited speakers and a hackathon wasn’t enough, we went the extra mile and had some awesome special guests drop in on us via videolink.
Kevin Mack, esteemed visual effects artist and Oscar winner, made a huge impression. Influenced by his comic-book-doused childhood, working in VR for him is a dream come true, being the ideal way to channel his artistic goals. He explored the union of music and VR, explaining how he had experimented with music and visuals throughout his life. His art is about inspiring awe, and he thought that music contributes uniquely to the visual experience. Therefore, it was important that music be given greater focus. He introduced his current work, which is an adventure into a crazy virtual world, shooting out beyond normal reality. The effect was totally inspiring — brilliant stuff!
Guy Whitmore, although not with us physically, was a huge presence in the room virtually. He talked about his philosophy on adaptive music, such as why linear music won’t do moving forward, and how an adaptive art can be achieved creatively and technically. As a composer working in interactive media, being able to think in smaller music chunks is vital, because the compositional fabric must be fitted together into larger wholes on the fly. Enabling music that can be dynamic or deeply adaptive is thus a unique skill. Guy surveyed the old divisions between between composer, performer and producer, once relatively clear cut but now no longer relevant in an adaptive context. Although, he admitted that he had previously overestimated the progress of adaptive music. Ten years ago he’d thought that we’d be way further along the road of adaptive music than we are now. The basic problem is that games companies aren’t yet fully behind this artform (although that’s about to change). Often they still want something big, cinematic and film-like, rather than something adaptive, which takes time and effort to pull off. Adaptive music requires more research, which in turn requires courage in exploring the new.
While the talks by our delegates were quite wonderful, the heart of a hackathon is in the hacking. And we had some super attendees, qualified and unqualified, experienced and inexperienced. Melodrive went all out with prizes for the event, with the generous support of our sponsor, Imagga. We’d like to thank them for fitting the bill and providing a great video promotion. The team winning best hack, Identity Crisis, was given an Oculus Go and 250 Euros, as well as a pro Imagga account for a year and t-shirts. The characters in the team were Alexander Grobe (VR developer) with Xingcen Zhou (developer) as second-in-command, and a dedicated team in Topper Sherwood (writer and journalist), Pola Weiss (psychologist) and Jomi Dadzie (student). Nathalie Raedler (singer/songwriter) also volunteered to provide some voice acting.
The team was exemplary in the speedy development and implementation of ideas, which involved creating a VR space that depicted the neural path inside a child’s brain. It depicted the internal events of the child experiencing a life-threatening accident, recalled through her interaction with a ‘real-life’ doctor at hospital. The use of a fully-fledged professional writer, Topper Sherwood, gave the hack super-depth, adding originality and edge. They incorporated Melodrive, using its ambient style for the overall soundtrack. This was coordinated with the ‘internal’ experiences of the girl, using Melodrive to match the music with her emotions. The effect was Dr. Mesmer on acid, inspiring everyone through its immersive and dramatic impact.
Okay, that’s enough from me, I’ve talked enough. I’m pretty sure that our guest speakers and hackers really enjoyed the experience of #EmoJam2, but let’s hear what some of our cool contributors actually said:
I came to the EmoJam, wanting to learn more about VR and its tools because I’m exploring their application in creative story-telling, including the telling of nonfiction stories. To me, EmoJam was valuable as a practical workshop, giving a diverse group of strangers the opportunity and means to work together to conceive and create an actual, if experimental, product. This non-competitive workshop allowed us to apply our existing skills, simply to learn about and experiment with this new technology. For what it’s worth, this is the most accessible work I did using this technology. – Topper Sherwood, writer and journalist
EmoJam2 was just awesome. A lot of things to learn, a lot of things to try. Hacking the VR/AR things was very close to going into “The Matrix”! Many thanks for organizing such a cool event! Looking forward to doing it again next year as well! – Dmitri Malisev, developer
The EmoJam provided a great opportunity for people to team up and bring an idea to fruition. The immersion of VR depends greatly on music, apart from the gameplay itself. At EmoJam we had the chance to integrate AI-generated music in real time in VR. – Xingcen Zhou, developer
It was a great experience to see so many people with different professions coming together. The jam was a new experience for me and I had a lot of fun because I wasn’t working alone, but in a team. I think this was a really good idea because you might end up doing things you’d never expected to do. Overall, I felt the atmosphere was so welcoming that I had a general good feeling being there.. -Jomi Dadzie, student
We’d like to thank our hackers for attending and making great experiences within an experience at #EmoJam2. We’d also like to thank our sponsors, Imagga and Berliner Pilsner, as well as The Venue Berlin for the use of the premises over the weekend — we’re super appreciative guys!