This week, I had the opportunity to see a unique musical act in Mannheim, Germany. Specifically, the one-man show put together by up-and-coming jazz phenom, Jacob Collier. Jacob Collier, for those that don’t know, gained notoriety through his creation of multi-part YouTube videos for which he not only composes original and creative arrangements of classic songs, but also plays every instrument in the mix. His style is characterised by extensive use of vocal layering/a cappella arrangements, versatile and extensive re-harmonisations, polyrhythms and metric modulations, all rooted in jazz/funk style inspired by the hit composers of previous decades (Stevie Wonder, Herbie Hancock, Michael Jackson, the Beatles, etc.).

He has a voracious appetite for music listening. From his interview with M Magazine: “when I was 18 to 19 I was at the Royal Academy of Music for two years. Every day I’d listen to an album on the way in and on the way home. That’s like 1,500 albums. I just kind of drank them in. From pop music to classical stuff and music from India and Africa. That was something I felt really helped me learn”. And it’s evident in his music. He isn’t afraid to borrow from any genre and make it his own.

“Every day I’d listen to an album on the way in and on the way home. That’s like 1,500 albums.”           –Jacob Collier

So why is his show of interest to a company building an AI composition system? Coming from a new generation, Jacob embraces technology with more enthusiasm and open-mindedness than most artists. In his debut album, In My Room, he was the only performer on the entire album, and thus he ran into an issue when he wanted to take his show on the road: he can’t play every instrument at the same time. Luckily, a fellow music tech expert, Ben Bloomberg from the MIT Media Lab, was inspired by Jacob’s video and had already reached out to offer his expertise in designing a live performance music system specifically for Jacob.

Innovation in Hardware: the Harmoniser

In 2014, Jacob and Ben worked together to push the technological limits of what can be done in a live performance setting. Through the “Imagination Off The Charts” residency at MIT, they experimented and implemented a variety of performance tools. First off, Jacob needed to replicate the characteristic vocal layering that he used throughout his videos. There are a few types of musical devices that can imitate that particular effect: the Vocoder or the Harmoniser. Vocoders transfer the pronunciation and spectral shape of the input voice to synthesized notes that are controlled with a keyboard, giving a robotic sound. Harmonisers, on the other hand, will pitch-shift the input audio to the frequency of the notes being played, so the audio input is actually kept mostly in tact. As Bloomberg mentioned,

The first thing we did was to build a harmoniser that could play twelve voice at the same time instead of four. Most people don’t know what to do with more than four notes at a time, and [Jacob] wanted twelve – that’s some pretty complex harmony”.

Indeed his harmoniser (along with his voice) is the centerpiece of his one-man-show. For a demo of the Harmoniser by Jacob himself, check out his Youtube video on the topic:

Not only is their Harmoniser an innovative device that expands beyond the harmonic capabilities of regular Harmonisers and humans alike, it also directly controls aspects of the visual performance. As Bloomberg explains in the aforementioned article: “for every key he pushes down, it sends a message to the video system, and creates a cutout copy of his head on the screen. So he can generate a virtual video chorus with up to 12 heads singing the different notes.” One can see an example of that in his live-looping performance of Don’t You Know. As the video below shows, four separate cutouts of Jacob’s head pop up on screen behind him when he first begins singing (as he also points to the sky in his characteristic fashion). At first it seems like the cutout placements are static, because he’s playing a static harmony on the keyboard. However, one can see clearly that the visuals are reacting to the harmony when hits the same note for the 2nd time.

Innovation in Live-Looping

In a normal live-looping show, the performer will be constantly returning to the device that is controlling the loops, in order to start and stop the recorded loops as they’re being played. What was unique about Jacob’s show was that you never once see him pressing a button or stepping on a trigger to control the looping. This was part of the goal of the live performance system he and Bloomberg built together. Bloomberg illustrates the difference between their system and a typical live-looping system: “it’s a complicated set of synchronized, choreographed systems, and events, and experiences, but it’s actually tied back to the human performance. It’s not on a timer. It’s not where somebody pushes play. It’s using sensors. It’s using interfaces that are so nuanced with instruments to extend the performance and the expressivity.”

At 1:38 in the above video, we can see an example of one of these potential sensors in action. In a transition into the very first chorus, Jacob simply walks over to the bass, and directly before the downbeat of a new measure, he plays a slide (or glissando) on the bass. The system must be logically conditioned to change to a new section upon new input from the upright bass instrument. All of the instrumental tracks other than the bass change from the previously looped content to entirely new, and previously recorded backing tracks. This isn’t to say that Jacob is ‘cheating’ by any means–the transition into an entirely new section is simply a limitation for any live-looping artist. One can simply not play a full arrangement at once, so if the artist wants the currently-sounding instruments to each play a new part, they’ll have to rely on backing tracks. The second time this happens, at 3:07 it’s the same thing. Jacob throws a quick glissando in on the bass right before the downbeat of the chorus section.

A similar thing may be happening towards the end of the chorus (and following section), when Jacob hits the crash cymbal with his hand–again a sensor listening to the crash could be triggering the next section, with a new set of pre-recorded tracks for each instrument. It happens at both 2:15 and 3:43.

There is an additional thing of note in the clip starting at 3:00. Before Jacob walks over to the bass, he can be seen changing the settings on his harmoniser keyboard. Why would he be furiously manipulating the interface of the harmoniser right before he leaves the instrument to play the bass? Isn’t he done with the harmoniser for the moment? The answer is that the harmoniser is still in action during the chorus–however it’s likely shifting the voices by a static interval. The harmoniser no longer has the input of Jacob’s fingers to decide exactly what notes to which his voice should be pitch-shifted, so he’s created presets to shift the extra voices relative to the pitch of his voice (likely octaves or other harmony-agnostic intervals).

Now not all transitions are controlled by sensors listening to particular instruments. For example, when he’s vamping on the piano at 4:46, there is another transition in backing tracks. It would require a much more sophisticated sensor to identify a particular sequence of chords on the piano–which is the only musical cue that happens–and to trigger a new section based on that. Both of the other examples of sensors initiating transitions were based on whether there was any input into the instruments–the instruments were either being played or they weren’t. In this case, it seems that the backing tracks are simply designed with a pre-recorded transition inside them.

Innovation in Musical Communication: Sonic Bloom Mountain

Even with the innovations made in hardware and live performance systems, Jacob and Ben were not finished. In a first-of-its-kind performance, they created a new way for musicians to communicate in realtime. In their performance with the MIT orchestra (below), they used a midi keyboard to automatically send the notes that Jacob played to mobile devices that were used by the other members of the orchestra, so that the musicians were playing what Jacob composed in realtime. Bloomberg elaborates:

“We’re thinking about ways that Jacob can play the ensemble sort of like an organ where they’re improvising in realtime reacting to things that he’s sending them on mobile devices. The idea essentially is to allow him to play a large ensemble like he would play the harmoniser, like he would use the looping system. But we have a million crazy ideas. There’s unlimited things that we want to try to figure out how to do. As of now he can send fragments of music and articulation to the phones of everyone in the ensemble. We have to play with it to see what that will enable musically.”

Skip to 2:53 for an example of the new mode of communication in action with the horns section.

Innovation in Action: The Show

The innovation doesn’t just stop in the design and implementation of new hardware and software systems. Jacob also experiments extensively with his performance systems. He creates an incredible effect with his harmoniser that he showcased towards the beginning of this particular performance. Utilizing four or (often) more simultaneous voices from his pitch-shifted harmoniser, he created these weaving glissandos which contrasted with the steady pitches of the currently-sounding chords. At first it was confusing as to how he could produce the given sound–was he using an oscillator on one of the voices? Perhaps he was using the bend bar to bend only a single of the harmoniser’s pitch-shifted voices. It was always only one of the voices that was doing the glissando, while the rest remained at a constant pitch. Therefore, I concluded that he must have been creating the long fluid pitch jumps and drops with his own voice, and harmoniser was still able to sample and pitch-stabilize the notes that he was playing with his keyboard. Jacob did an amazing job at utilising this technique to create a new texture, as he would abruptly switch registers as he changed the underlying harmonies, so as to obscure the source of the sweeping sound. The combination of the timbre of his voice and his almost android-like pitch precision really lends itself to the harmonising effect, and creates a very coherent texture.

With such a powerful tool at his disposal (the harmoniser), it would be easy for Jacob to spend all of his time simply creating his characteristic harmony-laced melodies, but he gives plenty of moments for raw performance. In his rendition of Close To You, for example, he starts the song with a minimal backing track, and simply his voice and the electric bass, relying only on his extensive musical chops to lead the show.

Photo by John Watson ©

Jacob uses a mic headset, and at certain points he reminded me of certain other famous musicians that use headsets to focus on their dance routines. Jacob certainly was dancing across the stage, jumping from instrument to instrument, keeping up with his looped tracks, often reaching to the next instrument while still playing the current one, and, at times, literally dancing. His ensemble of instruments spanned from the piano to the keyboard, the harmoniser, a set of handheld percussive instruments, a full drum kit, to an upright bass, an electric bass, an acoustic guitar, a melodica, and of course, his highly-tuned voice. At certain points (like with those aforementioned dancer/singers) you wonder if the vocals that you’re hearing are even his, because he’s running around stage, singing along with any and every one of his instrumental performances (including drums), and the vocal mix is perfect every step along the way. However, time after time with his selected vocal embellishments, his beatboxing, and his general improvisational style, there’s just no way it could be anything but live.

Almost as extraordinary as his musical talent is his pure positivity and performing confidence. At one point he went on a bit of a monologue to express his personal mission–to share intimate personal spaces and to spread little moments of joy, on whatever scale possible. During the performance, it did indeed feel that I was catching a glimpse of him in his own personal space, as if I was witnessing years of his committed rehearsals in his music room at home, watching him as he experienced the joy of discovering a new chord or mastering a polyrhythm. Towards the end of the show, after over an hour of constant singing, jumping, slapping the bass, endless keyboard solos and drum grooves, he isolated his voice and live-looped a 6-part harmony, pushing the limits of his own vocal ability to find a voice range that was unoccupied in the current mix at each successive loop. Only with his incredible performing confidence and optimism could he take such risks, and only with his deep instrumental skills could he pull them off. It’s obvious as a spectator that he wants to share as much as possible with his crowd–to create meaningful connections with people on whatever scale he can. And what’s more, he embraces technology with the same optimism and open-mindedness as he embraces his audience.

As a final note, anyone who plays a melodica solo in the last song of a set is a friend of mine.

Image credit: The Guardian

Title Image credit: