Ever wanted to do more with the music or SFX in your game? Maybe you want to go beyond triggering audio clips with basic effects towards infinite variations of explosions or gunfire? Maybe your player characters are robots and you want to vocode the player’s microphone input? Perhaps you want complete playable instruments within your game, or unique melodies composed for each user-generated character a la Spore?

If so, then using Pure Data (Pd for short) may be just what you need. Sure, you can do a lot of these things using FMOD and Wwise, but Pd makes the process so simple and elegant, and best of all: it’s free. If this sounds like your cup of tea, then read on!

In this tutorial I’ll be showing you how to include Pd (well, actually libpd, but more on that in a bit) as a native plugin within Unity. You’ll learn how to send messages and data to and from Pd, and how to use those messages to create an algorithmic melody. We have the finished project with everything included on GitHub if you’d like to download that and skip to the end.

You don’t need to know anything about Pd or games audio to follow this tutorial. We do assume some basic Unity and C# knowledge. Before we get started I recommend you download Pd if you haven’t done so already – we’ll be editing some Pd files so you’ll need it a bit later on.

0) What is Pure Data and why should I care?

Firstly, let’s get some terminology straight. Pd is an open source audio synthesis and processing application. It has a very visual way to “program” sound, by connecting little boxes together with virtual wires.

Libpd is Pd’s embeddable library that lets you use Pd’s audio engine in pretty much anything from desktops, to mobile phones, to Raspberry Pis.

At Melodrive, we’re using libpd for its ability to sonify our adaptive music in-game. It’s lightning fast processing time, cross-platform compatibility, ease of use, and permissive MIT license are of course big bonuses!

1) Getting started

We’re going to be using an excellent starter pack, created by batchku called libpd4unity-starter. This is based off patricksebastien‘s work on libpd4unity, but contains everything we need to get going super fast. Go ahead and fork it, clone it, or download the zip from GitHub.

Note:
You need to be running Unity 32-bit for this all to just work. Running on 64-bit is possible, but you will need to compile the libpd DLL yourself in 64-bit (or grab one of the pre-built versions). For ease, I’m sticking with 32bit. If you’re seeing this message:

Unity 64-bit DLL error

Followed by a bunch of “DLL not found” messages when you click play – this is the reason!

Opening the project

Open the libpd4unity-starter project in Unity.

If you’re using the latest version of Unity, you’ll be asked to upgrade the project to the newest version. Go ahead and do that now – it doesn’t break anything.

When the project opens, you might see some errors in the console:

mcs error

Since libpd deals with pointers, it requires an ‘unsafe’ context in C#. To allow this, an “mcs.rsp” file is needed in the Assets folder containing the line

-unsafe

This adds a flag to the C# compiler options when compiling your project. libpd4unity-starter includes some deprecated files and we need the new “mcs.rsp” file. Simply duplicate the “gmcs.rcp” file and rename it to “mcs.rcp”.

Restart Unity and these errors should go away.

Click play and you should see a ball bouncing up and down – but more importantly you should hear some whistling wind-like sound and a crunch whenever the ball hits the floor.

libpd4unity-starter project

2) Understanding the project

Now that we have Unity making some sound with libpd, let’s have a look around the project to find out how this is happening.

In your Scene you should see a GameObject named “LibPd” – this is the main interface between Unity and libpd. It is set up to be an Audio Listener, which means it receives input from any given Audio Source in the scene and plays sounds through the computer speakers. Also attached to the LibPd GameObject is the “LibPdFilterRead” script – this is the main interface between Unity and libpd, so let’s open this script now.

LibPD GameObject

LibPdFilterRead

If you come from an audio background, you might think that LibPdFilterRead operates like an audio filter. It can certainly do that, but it can also do a lot more. Instead, think of it like a generic audio input/output. The OnAudioFilterRead method takes a float array and can manipulate it in any way, adding audio data, using filters, adding effects – even completely replacing the audio stream (which is what’s happening here).

In LibPdFilterRead, the OnAudioFilterRead method calls the following Method in the LibPd object:

public static int Process(int ticks, float[] inBuffer, float[] outBuffer) { ... }

This method takes an input buffer, processes a number of “ticks” worth of data, and writes that to the output buffer. If you’re wondering what one “tick” is – it’s usually 64 samples of audio data per channel (so that’s 128 samples in stereo). You can increase the number of ticks to balance audio performance at the cost of latency. In LibPdFilterRead, this is calculated based on the target platform and user options.

Whatever is left in the data array after the various OnAudioFilterRead methods have been called is what’s sent to the speakers. By using float[] data as both the input and the output buffers of LibPd.Process, the audio will be completely replaced by whatever libpd outputs.

The next question to answer is: how does libpd know what to do when processing?

You’ll notice that the LibPdFilterRead script has one public variable:

public string nameOfPatch;

In Pd your audio code is collected into files known as “patches”. Libpd can in fact open more than one patch at a time, but in this starter project only one main patch is specified. Whatever is in this patch is making the audio we’re hearing. In the unity inspector we can see that the Pd patch is called bouncingBallSound.pd. This file is located in Assets/StreamingAssets/PdAssets. Let’s have a look at the bouncingBallSound patch now.

3) Understanding bouncingBallSound.pd

When opening this file in Pd, here’s what you should see:

bouncingBallPatch.pd

OK, so it might look confusing (especially if you’re unfamiliar with Pd), but this is a very simple patch – honest! Each box (object in Pd terminology) is like a function in programming, they can have inputs at the top (known as inlets) and outputs at the bottom (known as outlets). Inlets and outlets are shown as thick black bars on each object.

By the way, at any time you can hear what this patch sounds like by turning on the DSP in Pd – there’s a little checkbox in the top right of the main window, for you can select Media > DSP On.

Let’s go through the basics of what this patch is doing.

At the top left of the patch you’ll see a [loadbang] object. This outputs a triggering signal (known in Pd as a bang) when the patch is loaded. This is used to kick things off in the patch and is similar in a way to a GameObject’s Awake method.

If you follow the wires down you’ll eventually come to a [metro 20]. Metro is short for a metronome, after it’s received a bang in its inlet, it will send out bangs every 20 milliseconds. You can use these objects for timing.

Moving along the wires there are a few more objects which we’ll ignore for the time being, suffice to say that they pick a new pitch whenever they receive a bang (so every 20ms in this case). Eventually we reach two [osc~] objects. Osc is short for oscillator and the little tilde (~) after the name means that this is an audio signal. It is these [osc~] objects that are making the low-pitch whistle you can hear.

Following the wires further down we eventually come to a [dac~] object. This stands for digital to analogue converter, and is the audio output of the whole system. It has two inlets for left and right channels, so whatever signal comes into here is passed on to the float[] outputBuffer mentioned earlier. You’ll see that each inlet is taking input from it’s own [osc~] object.

Editing the patch

To recap, the left side of the patch is where the wind-like sound is made. Let’s try changing the values in Pd and listen to the result.

First try changing the [metro 20] to something slower. To do this you will need to enter “Edit Mode” (Edit > Edit Mode or Ctrl+E). Double click on the object and change the number 20 to something like 1000. Now exit Edit Mode and click the toggle object twice just above (the thing that looks like an [X]). You should hear the wind sound change – the pitches are changing at a much slower rate. Congratulations, you’ve just made your first algorithmic music!

Now let’s see how to make this sound much quieter, just before [dac~] you’ll find two [*~ ] objects. The signal from the oscillators is being multiplied here. A number between 0-1 will reduce the volume, any number > 1 will increase it. Double click them (remember to be in Edit Mode) and change both these numbers to be 0.1 instead of 0.5. You should notice an immediate drop in volume as soon as you click out of the object.

Now let’s draw our attention to what’s going on in the right side of the patch. I’ll explain the [r bounce] in a moment, but for now look just under at the circular button. This is an interactive “bang”, make sure you’re not in Edit Mode, and click it. You should hear the familiar crunch of the ball bouncing.

There’s actually three parts to this sound: an oscillator, some noise, and an envelope. We won’t worry about the noise and the envelope for now, and focus on the oscillator. The oscillator is a very low frequency (50Hz) so you probably can’t really tell it’s there. Try setting a higher frequency, 440Hz, and it should become more obvious. To do this, once again make sure you’re in Edit Mode, and double click the [osc~] object. Change the number 50 to 440 and click outside the object. Now exit edit mode and click the bang to trigger the sound. You should hear a much more obvious “beep”.

Save your patch and turn off the DSP in Pd before you’re driven mad by the random pitches!

4) Sending messages

By now you should have a basic grasp of how Unity and Pd are working separately, but there’s one final piece to the puzzle. In our scene, the ball is triggering the crunch sound whenever it bounces. This is achieved by sending a message to libpd, which triggers the bang object we were playing with in the last section. Let’s find out how this is done.

In the Unity scene, inspect the Sphere object.

Sphere inspect panel

If you’re Unity savvy a lot of this stuff will be familiar to you. This component we’re mainly interested in is the BounceToPd script. Open this script now and scroll down to the Update method. The script is doing a basic check on the vertical velocity of the ball. When the direction changes from positive to negative we know that the ball has bounced. Line 35 contains the vital code we’re interested in:

LibPD.SendFloat ("bounce", 1);

This is a message to Pd that sends some data over as a float. We have an identifier and the actual value we want to send. Hopefully the identifier looks familiar – in bouncingBallSound.pd do you remember seeing the [r bounce] object? The “r” is short for receive; it’s here where the message ends up and will pass the float on down it’s outlet wire. In the patch right now this message is simply converted into a “bang” message, so the value is lost.

There is actually a second message that is being sent in this script. Can you tell what it does? (There is a comment in the Pd patch that will give you a hint).

Any number or string can be send to Pd and received back in Unity through similar messaging methods. Complex objects such as Arrays can also be sent.

5) Getting more musical

Let’s make a little melody as the ball is bouncing by using the LibPD.SendFloat call to change the oscillator pitch whenever there’s a bounce.

First, in C# we’ll write some code to generate our melody:

private int[] scale = { 0, 2, 4, 5, 7, 9, 11 };
private int root = 60;

int GetPitch () {
    int i = Random.Range(0, scale.Length);
    return scale[i] + root;
}

The new GetPitch method will return a random pitch from a C major scale. These pitches are represented as MIDI note numbers – the de facto standard used in almost every electronic instrument.

Now we modify the Update method to get this pitch and send it to Pd:

int pitch = GetPitch();
LibPD.SendFloat ("bounce", pitch);

Instead of sending a simple 1 float all the time, we should start sending different pitches.

Switch over to bouncingBallSound.pd in Pd and make sure you’re in Edit Mode. Click Put > Object (or press Ctrl+1) and place an empty object a little above the ball’s [osc~] object. In this new object, type “mtof” and then click anywhere outside of the object to make it. [mtof] converts the MIDI note into a frequency in Hz (e.g. MIDI note 69 is 440Hz).

Connect the outlet of [r bounce] to the inlet of [mtof], you need to click on the black bar to do this (the mouse should change pointer to indicate this). Connect the [mtof] outlet to the first inlet of [osc~]. Remove the [noise~] and [*~ 0.25] objects any selecting them and pressing Delete.

The right hand side of your new patch should look like this:

Pitch control on the bounce

Save your patch and test it out in Unity’s play mode. This sounds OK, but could be better!

Change the [osc~] to a more interesting sound, a [phasor~], and add a [-~ 0.5] after it, like so:

[phasor~] object

Finally, we’ll add a reverb to the sound to make the space sound more cavernous.

First, disconnect all output wires from the [*~ 0.5] object (not the [-~ 0.5] connected to the [phasor~]) by selecting them, and pressing Delete.

Now create a [rev3~ 100 90] object just below that. Connect the [*~ 0.5] outlet to the first inlet of [rev3~ …], and the first outlet of [rev3~ …] should go back to your [dac~] inlets.

Your finished patch should look like this (I’ve tidied it up a bit):

The complete bouncingBallPatch.pd

There’s one more thing required to get the reverb to work: libpd is designed to be super lightweight, so [rev3~] will not work straight away, even though it’s built into Pd. You need to copy the rev3~.pd file into your Unity project.

The easiest way is to right click [rev3~ …] and click Open. From this window, click File > Save As and navigate to the Assets/StreamingAssets/PdAssets folder of your Unity project. Save it here.

When this is done, click Play and you should hear a much more interesting bounce melody!

6) Round-up

In this tutorial I showed you how to hook up libpd with Unity. We went through a pre-made starter project and familiarised ourselves with how the whole system is set up and how data and messages are passed between the two systems. Finally, we edited both the C# and the Pd patch to make the bouncing ball play a unique melody every time the ball bounces!

The finished project with everything included is available on GitHub. If you have any questions, feel free to comment below or on GitHub!

We’ve barely scratched the surface of what’s available in Pd, but I hope this tutorial has inspired you to consider using it in your next game. What are you planning to use Pd for? Have you released any games or apps using it? We’d love to hear from you if you have!