grantheus.com


About
News
Contact

Post Production Audio
Video
Photography
Visual Artwork
Sonified Chess
DIY Cello
Double Jean DJ


backdrop: sonifiedchess1
© 2025, Grant Heus

Sonified Chess

My brother and grandfather playing Sonified Chess at Columbia College Chicago’s end-of-the-year showcase Manifest 2023


During my Spring 2023 semester, I took a course called Audio Programming and Performance. We were taught as a laptop ensemble and encouraged to explore sound design within the context of coding, as well as how we could create music interfaces in a fun and creative way. My idea for Sonified Chess started in class and stemmed from the realization that a chessboard consists of a balanced binary of black and white squares. If I could somehow represent this balance as a value, I knew I could set change thresholds based on a chess piece’s position.

First, I had to decide how I wanted to read the chessboard. I knew that choosing a virtual chess game would give me more options down the line. For instance, mapping specific sounds to specific pieces or moves, or even implementing machine learning to map parameters such as game-winning probabilities might be easier to do by capturing a virtual game of chess on a computer. However, I am still fairly new to coding, and I wasn’t as concerned with having a program that interprets the game of chess itself. I wanted to create a tangible musical interface that people could interact with. This way, my instrument could explore the relationship between playing chess and performing—how does hearing what I’m doing on a chessboard impact the way I play the game with performing music in mind? Initially, my plan was to read four separate sections of the board to create chords. Each quadrant of the board would be set with a different note as a base. In my testing, I used a C major seventh chord, assigning C to the first quadrant, E to the second quadrant, G to the third quadrant, and B to the fourth quadrant. My hope was that I would be able to change these notes within a scale as the balance of the black and white changed with the position of the board.

Flowchart for visualizing the logic of my code along with the board I built out of a thrifted chess board and scraps


As a result of my desire to create a product people could interact with, I chose to use an external webcam with a black-and-white threshold filter set above a physical chessboard. In putting the webcam through a threshold filter, I essentially create a binary of two pixel values—black or white. This binary is especially effective for reading a chessboard since squares and pieces are commonly distinguished by being black or white. Using the “get()” function of Processing, I was then able to extract the number of white pixels in this feed. This value acts as the “balance value” I spoke about earlier. My program only needs the white pixel value because it knows the total pixel range it’s reading, and because there are only two types of pixels, the “balance” of the area with just one of the two values. As a game of chess is in progress and the board positioning changes, so will the white pixel value on any given part of the board. For example, if a white piece moves onto a black square the white pixel count goes up. Inversely, if a black piece goes onto a white square the white pixel count goes down. The program loops a read of the number of white pixels in each quadrant of the board for each frame and makes changes to notes playing based on conditional thresholds I set in my code. These change thresholds are tuned to the number of pixels that changed based on a piece being somewhere else, which averaged around 650 pixels. 

Messages are then sent via OSC to SuperCollider where they can be interpreted for a wide variety of applications. For instance, I started by sending note frequencies to change the frequency argument of a SynthDef. This was an easy process, as I simply set conditionals to send specific frequencies as my OSC messages, which would get received into SuperCollider and set to the variable that was controlling the frequency of my instrument. In replicating this process for each quadrant of the board, I was able to accomplish my initial goal of playing chords. I took this idea a step further by sending triggers for pattern definitions (Pdefs) instead of frequencies. In this scenario, each quadrant of my chessboard is set to a specific instrument. For instance, my first quadrant might be a kick drum. Instead of changing notes, I can change the sequential pattern that my kickdrum is playing by redefining it with the same thresholds I’ve set for note change. By setting each quadrant of the board to a different instrument, my chessboard is now able to create entire song structures based on what is happening in the webcam feed. I found that this specific program was fun for me, but less comprehensive for my audience.

My favorite application was a program I wrote for the chessboard to send MIDI information out of my computer via USB. Working with MIDI opened a world of possibilities in terms of using my chessboard musically. I mainly sent MIDI out into my Roland JX-08 synth module. There, I could send my notes through the two separate patches on that synth, as well as run them in parallel to a step sequencer. My favorite way to demonstrate how this can be musical is by setting the first patch to receive my chessboard notes and running the second patch as a bassline in the sequencer. I am able to sync the tempo of the sequencer with the tempo of my chessboard with a formula I set in my code to set BPM for my webcam’s framerate. In feeling out the rhythm of the chessboard, I can run the sequencer and sync the tempo out of my synth and send that tempo out via CV to other instruments, such as a drum machine or even another synth. However, this solution isn’t perfect, and there are limitations. One issue is the webcam—the framerate becomes inconsistent when run at an exponential value. Programs run best with framerates that are whole numbers such as 2 or 4, which sync well to 120 bpm.

Building this project was a significant learning experience for me. I would often get stuck and have to problem-solve around issues. I learned that, for me, often the best approach to troubleshooting was to follow the logic I had laid out in my planning and to pass printed messages as pseudocode to get my code to theoretically work. It was also helpful to turn to my peers and professor when I was having trouble. Often, having conversations with them would make me think about my code in a new light or cause me to consider solutions I wouldn’t have thought of on my own. Building devices such as this chessboard brings together everything I love about being an artist: creative problem-solving, working within limitations, planning, and structuring. I loved seeing my idea come to life, and I look forward to using what I’ve learned about programming to continue to build musical interfaces.

In addition to taking Audio Programming and Performance that semester, I took an Intro to Poetry class. On the surface level, this may seem unrelated, but much of what we learned in Introduction to Poetry informed my newfound understanding of coding. Much like a poem, code is a creative expression; programming reflects the logical thinking of an individual. There are often many different ways to structure code and many paths to the same result. For instance, in this programming class, we were all assigned a project to build code based on the same prompt. To everyone’s surprise, we all came in with extremely different programs the next week. It was interesting seeing how others approached the assignment with different thinking. These ideas are mirrored by poetry in elements such as poetic forms, syntax, meter, and literary devices. All of these elements—when structured in the correct order—function to communicate and accomplish a set goal. This ultimately underscores how creativity and structure work hand-in-hand, which is an important lesson for artists of any medium.

My biggest takeaway from this project was gaining an understanding of the endless scale and potential coding has in sound design. Coding and electronics as possible avenues for me to explore my voice as a sound designer. Even now, I have continued to brainstorm ways in which I could take my chessboard further, as well as the possibe new projects I could start. I am energized by programming and plan on creating more projects in the future.