Thesis Week 11/12: Spatial Sound, an Apple Watch App and a Beating Heart

In the last weeks of thesis I iterated one more time with the agent simulation and how we as humans can perceive autonomous AI: I finally settled on sonifying the experiences of the agent in a quadrophonic speaker setup. Music and sounds are strong emotional components in experiences, focusing on them without showing a screen with the simulation creates a stronger bond to the AI agent walking through its paradise. As music affects our feelings directly, in a way your heart “visualizes” the simulation better than your eyes:

“And now here is my secret, a very simple secret: It is only with the heart that one can see rightly; what is essential is invisible to the eye.”

(Antoine de Saint-Exupéry, “The Little Prince”)

Therefore the audience can hear exactly what the AI hears in real time, the simulation of paradise is mapped to a real 3D space. Each element in the unity simulation emits a unique sound the moment the agent sees it (using raycasting for detection of surroundings). Unity translates this sound into a surround sound experience as the agent moves through the simulation environment. The audience listens to it through an audio listener situated on the agents head. While listening the audience is sitting in a physical booth with 4 speakers in each corner of the room. The audience chair is placed on an inflatable boat to create a third space and the experience of floating through sounds.


To create an even more intimate connection to the agent, the agent can only explore its paradise if there is human life in the installation: I wrote a little Apple Watch App that detects the heartbeat of the audience (or artist) in realtime, sends it via a node js server to the unity server (using a simple RESTful API I set up) and ties the steps of the agent to it. As an extra feature and control mechanism the watch displays the detected element in the simulation as well, just as a simple word, for example “birds” (useful for the artist, distracting for the audience though - something to improve …).


Here an overview of the tech setup:


And a short video: