Wolfram Blog
Alec Shedelbower

How I Built a Virtual Piano with the Wolfram Language and the Unity Game Engine

June 6, 2019 — Alec Shedelbower, Kernel Developer, Algorithms R&D

How I Built a Virtual Piano with the Wolfram Language and the Unity Game Engine

You know what’s harder than learning the piano? Learning the piano without a piano, and without any knowledge of music theory. For me, acquiring a real piano was out of the question; I had neither the funds nor space in my small college apartment. So naturally, it looked like I would have to build one myself—digitally, of course. And luckily, I had Mathematica, Unity and a few hours to spare. Because working in Unity is incredibly quick and efficient with the Wolfram Language and UnityLink, I’ve created a playable section of piano, and even learned a bit of music theory in the process.

First, I determined that building the piano requires the following:

  • Audio for each musical note
  • Geometry for the piano keys
  • A portable, interactive, real-time-rendering audio and 3D-physics engine

The first two can be accomplished trivially in the Wolfram Language. As for the last one, I opted to use the newly introduced UnityLink—a powerful link between the Wolfram Language and the real-time development platform Unity. Using UnityLink, it’s now possible to combine the advantages of the Wolfram Language’s impressive simulations with regards to rendering, audio and physics, with Unity’s efficient packaging of all three into standalone applications for web, desktop, mobile and console platforms.

What Is a Piano?

Before I dive into the code, let’s explore some of the background on the piano and the musical notes it plays. Understanding the theory behind the physical piano will help us to better recreate it digitally in Unity.

The piano traces its origins back to early 18th-century Italy, where it was invented by Bartolomeo Cristofori. Since then, it’s undergone many design changes, eventually resulting in a (mostly) standardized key configuration.

The modern piano has a total of 88 keys, 52 of which are white and are used to play the natural notes (A, B, C, D, E, F and G). The remaining 36 keys are black and are used to play the accidentals (A♯/B♭, C♯/D♭, D♯/E♭, F♯/G♭ and G♯/A♭). The ♯ and ♭ symbols stand for sharp and flat, respectively. Here you can see all 88 keys with their corresponding notes labeled:

Full piano keyboard

The notes can be further divided into octaves, each of which contains 12 keys. Two keys with the same note but in different octaves will have different pitches. The octaves of a piano are color-coded in this diagram:

Octaves

A piano contains seven full octaves, with four extra keys on the ends. These extra keys allow the scales of A minor and C major to be played in all seven octaves.

In this blog post, and for simplicity’s sake, I’ll focus on a single musical scale (ordered list of notes), but you can apply this method to create the entire piano. Let’s use one of the most common scales—the C major scale. This scale contains only the natural notes in the order C, D, E, F, G, A and B. Any C note can be chosen as the start of the scale. Here, I’m going to use the C note in the fourth octave (also known as C4 or middle C):

Fourth octave

If you take a closer look, you can see that this subsection of our piano contains all seven natural notes and all five accidentals. Note that I also included the C key from the next octave (C5) in the scale, as this helps “round off” the scale:

Middle C

Making Music

Whew! With the background out of the way, I can finally get to the code. To get the sounds of the piano keys I use the symbol SoundNote, which can generate any note from a large collection of instruments. For a single note, you simply give it the note name, duration and instrument. When wrapped in Audio, it creates an audio object that can be played directly in a notebook:

Audio
&#10005

Audio[SoundNote["C", 3, "Piano"]]

To get a note in a specific octave, you simply concatenate the octave number to the end of the note name. For instance, I can get all the natural notes in the fourth octave using the code shown here:

naturalNotes
&#10005

naturalNotes = {"C", "D", "E", "F", "G", "A", "B"};
Table[Audio[SoundNote[note <> "4"]], {note,
   naturalNotes}] // AudioJoin

Generating Geometry

The exact shape and dimensions of piano keys vary by manufacturer. I opted to keep things simple by approximating each key as a prism. The advantage of using prisms is that I only need to specify the base polygon and extrude upward. However, ensuring no keys overlap requires five base polygon variations:

Five base polygon variations

All that’s left is to convert the base polygons into 3D prisms. This can be done easily using RegionProduct to multiply the polygons by a line segment with a given height:

line = BoundaryMeshRegion
&#10005

line = BoundaryMeshRegion[{{0}, {height}}, Point[{{1}, {2}}]];
regions =
  Table[RegionBoundary[
    RegionProduct[BoundaryMeshRegion[polygon], line]], {polygon,
    polygons}];
Row[regions]

Preparing the Project

Now that I have the audio and geometry, it’s time to combine them in Unity to make a working piano. As I mentioned previously, this is made possible with UnityLink.

With Unity installed, loading UnityLink is as simple as a single function call:

Needs["UnityLink`"]
&#10005

Needs["UnityLink`"]

I start by opening a new Unity project, which I’ve named "MyPiano":

UnityOpen
&#10005

UnityOpen["MyPiano"]

MyPiano

With the project open, I can now send and receive data from Unity. I will eventually want to create my piano in a Scene—a 3D environment that can act as a menu, game level or any other distinct part of a Unity application. But before I create my Scene, I have to first transfer the audio and geometry content I created earlier to Unity. Once it has been added, I will be free to use it in my Scene.

While not required, it’s good practice to keep your Unity project organized with subdirectories in the project’s Assets directory. The Assets directory contains all of the assets used in the project (textures, audio clips, meshes, etc.). In the line shown here, I create five directories in the Assets directory using CreateUnityAssetDirectory:

CreateUnityAssetDirectory
&#10005

CreateUnityAssetDirectory[{"Meshes", "Audio", "Materials", "Scenes",
   "Scripts"}];

Asset directory

Now I go about transferring the audio. I do this by passing the Audio of each note to the function CreateUnityAudioClip, which automatically converts it to Unity’s AudioClip object and stores it in the Assets directory. These AudioClip objects are represented as UnityAudioClip expressions in the Wolfram Language:

notes
&#10005

notes = {"C4", "D4", "E4", "F4", "G4", "A4", "B4", "C#4", "D#4",
   "F#4", "G#4", "A#4", "C5"};
clips = Association[Table[
    audio = Audio[SoundNote[note, 2.5, "Piano"]];
    note ->
     CreateUnityAudioClip[File["Audio/note_" <> note], audio], {note,
     notes}]];
clips // Short

Audio

Next, I transfer the geometry of my piano keys. This time, however, I use CreateUnityMesh to automatically convert my MeshRegions to Unity’s Mesh objects, represented as UnityMesh expressions in the Wolfram Language.

meshes = Table
&#10005

meshes = Table[
   CreateUnityMesh[File["Meshes/mesh_" <> ToString[i]],
    regions[[i]]], {i, Length[regions]}];
meshes // Short

Unity meshes

I do something similar to create a black and a white material, as well as a script component for controlling the user interaction with the piano keys. I’ve left these out for brevity, but the full code can be found in the downloadable notebook for this post.

Setting the Scene

With all of the Assets transferred, I can finally make the Scene for my piano. I start by creating a new default Scene:

CreateUnityScene
&#10005

CreateUnityScene[File["Scenes/Piano"]]

If you’re new to Unity, here’s a brief description of Scenes. Scenes contain Game Objects, which in turn act as containers for Components. You can think of the Scene as an environment, Game Objects as the things in that environment and Components as the behaviors of those things.

In my piano Scene, I’m going to make a Game Object for each key. I’ll then attach the script component I created earlier to each of these game objects, so they make sound and move when the user interacts with them.

I could just add each key one at a time; however, that would prove to be tedious and difficult to extend in the future. Instead, I define the information about each white key and each black key in two lists. I can then iterate over these lists to create each key automatically. For each key, I specify the computer keyboard key it corresponds to, the musical note it should play and the index of the mesh it should use. Note that the mesh index for black keys is implicitly assumed to be 5:

whiteKeys
&#10005

whiteKeys = {<|"Keycode" -> "q", "Note" -> "C4", "Mesh" -> 3|>,
   Sequence[
Association["Keycode" -> "w", "Note" -> "D4", "Mesh" -> 4],
Association["Keycode" -> "e", "Note" -> "E4", "Mesh" -> 2],
Association["Keycode" -> "r", "Note" -> "F4", "Mesh" -> 3],
Association["Keycode" -> "t", "Note" -> "G4", "Mesh" -> 4],
Association["Keycode" -> "y", "Note" -> "A4", "Mesh" -> 4],
Association["Keycode" -> "u", "Note" -> "B4", "Mesh" -> 2],
Association["Keycode" -> "i", "Note" -> "C5", "Mesh" -> 1]]};
blackKeys = {<|"Keycode" -> "2", "Note" -> "C#4" |>, Sequence[
Association["Keycode" -> "3", "Note" -> "D#4"], Null,
Association["Keycode" -> "5", "Note" -> "F#4"],
Association["Keycode" -> "6", "Note" -> "G#4"],
Association["Keycode" -> "7", "Note" -> "A#4"]]};

To keep my Scene organized, I also group all of my keys under a parent game object named "Piano Scale":

parent = CreateUnityTransform
&#10005

parent = CreateUnityTransform["Piano Scale"] 

I iterate over all the white keys first:

Do
&#10005

Do[
 key = whiteKeys[[i]];
 name = "Key " <> key["Note"] <> " (White)";
 go = CreateUnityGameObject[name, meshes[[key["Mesh"]]]];
 go[["Transform", "Position"]] = {(i - 1)*(whiteWidth + gap), 0, 0};
 go[["Transform", "Parent"]] = parent;
 script = CreateUnityComponent[go, "PianoKey"];
 script[["Key"]] = key["Keycode"];
 script[["Clip"]] = clips[key["Note"]];
 , {i, Length[whiteKeys]}
 ]

White keys

This is followed by the black keys:

Do
&#10005

Do[
 key = whiteKeys[[i]]; name = "Key " <> key["Note"] <> " (White)";
 go = UnityLink`CreateUnityGameObject[name, meshes[[key["Mesh"]]]];
 go[["Transform", "Position"]] = {(i - 1) (whiteWidth + gap), 0, 0};
 go[["Transform", "Parent"]] = parent;
 script = UnityLink`CreateUnityComponent[go, "PianoKey"];
 script[["Key"]] = key["Keycode"];
 script[["Clip"]] = clips[key["Note"]];
 , {i, Length[blackKeys]}
 ]

Black keys

For each key, I create a Game Object with the appropriate mesh using CreateUnityGameObject. After setting the position of this Game Object, I attach the custom script I created earlier by passing the Game Object and script name to CreateUnityComponent. I finish by specifying the keycode and audio clip for that key.

And just like that, I have a working (partial) piano. However, it doesn’t look as good as it could. To remedy this, I adjust the object materials along with the lighting and camera (full code in the downloadable notebook). With this, we get the final result:

Final piano

Now that looks better! Before moving on, I also want to save all the changes I just made to my Scene by calling SaveUnityScene:

SaveUnityScene
&#10005

SaveUnityScene[]

Playing the Piano

To test the piano in the Unity editor, I can use UnityPlay and UnityStop to switch between the Play and Edit modes. When I’m satisfied with the results, I can build the project to a standalone application using UnityBuild.

The following command will automatically build the project to a file in my project directory for my current platform (macOS):

UnityBuild
&#10005

UnityBuild[]

With the build successful, I can immediately open and play my piano application:

SystemOpen
&#10005

SystemOpen[%["Application"]]

SystemOpen

One of the advantages of working in Unity is its ability to build to numerous platforms without having to change your code. If you can play a game on a platform, odds are that Unity can build to it.

It can even be built to run in a web browser. Go ahead and try it!

Your Turn!

This small section of the piano can easily be extended to a full piano keyboard. With more than 160 styles and percussions available in SoundNote, you could also build other instruments or even combine them into a single synthesizer.

To start working with UnityLink in the Wolfram Language, visit the online documentation page or try out one of the sample projects. There’s so much you can do with the built-in interface, and I look forward to seeing what projects you come up with on Wolfram Community!

Version 12 brings a host of major new areas into the Wolfram Language, including a seamless interface to the Unity game engine. Start coding today with Wolfram|One or Mathematica, on the desktop or in the Wolfram Cloud.

Get started!

Leave a Comment

3 Comments


Darren Liu

This is incredible! As a musician and long-time Mathematica user, I’m blown away by the versatility of the software and the creativity of the developers behind it. Keep up the great work.

Posted by Darren Liu    June 6, 2019 at 12:35 pm
Larry Adelston

Alec, this is awesome! Thanks for sharing your work, and for clearly describing your development process. Definitely food for thought.

Posted by Larry Adelston    June 6, 2019 at 2:28 pm
Sy

This is great. I wonder if I can try UnityLink with free Wolfram Engine.

Posted by Sy    June 7, 2019 at 12:22 pm


Leave a comment