invite users to contemplate on their online selves and identities by creating a semi-realtime chat with pre-recorded videos and actors - and the possibility for users to participate in this fake setup. 


  • how do we establish trust online with strangers?
  • how do we perceive ourselves in a group environment online?
  • what are the rules of communications in a video chat with strangers ?
  • how is time related to communication ?

inspiration / early sketches

We started with early sketches that were playing with video feeds in VR. Initially we wanted to give users the possibility to exchange their personalities with other users: We had the idea of a group chat setup where you could exchange "body parts" (video cutouts) with different chat group participants. This should be a playful and explorative experience for participants. How does it feel to be in parts of another identity? How do we rely on our own perception of our body to establish our identities? How do we feel about digitally augmented and changed bodies? How does that relate to our real body perception? Does it change our feeling for our own bodies and therefore our identities? How close are we to our own bodies? Do we belief in body and mind as separate entities? How is body and mind separated in a virtual and bodyless VR experience? 



After looking at our timeframe and technical skills we decided to postpone the VR element of our concept and focus on the core ideas in a 3D setup: body perception, trust and identity in virtual interactions. We chose three.js as our main language as it provided us with a lightweight environment in the browser that possibly could be deployed in an app or in a local environment. As we decided later for a local setup for our group online environment this proofed to come with a few tradeoffs regarding access to files on the local machines from javascript. We used python as backend tool to compensate the security restrictions in javascript. 


conceptual setup

Close to our initial sketches we constructed an online environment that should fulfill the following requirements:

  • group experience with person-to person interaction
  • video feeds like in a webcam-chat 
  • insecurity, vagueness and un-predicability as guiding feelings for the interaction with strangers
  • fake elements and identities to explore the role of trust in communication
  • a self-sustainable environment that has the potential to grow or feed itself

To achieve that we built a welcome page that simulates a video-chat network. The aesthetics were kept partially retro and cheap. The website should look not completely trustworthy and already invoke a feeling of insecurity - but still stimulate the interest in the unknown. 

The main page features 3 screens with the user's webcam feed in the center - right between two users that seem to establish a communication. The user should feel in-between those two other users and feel the pressure to join in in this conversation. Both users on the left and right are actors, the video-feeds pre-recorded and not live. The user in the middle does not know this - they should look like realtime chat-videos. 

While trying to establish a conversation with the fake video-chat partners, the webcam-feed of the user gets recorded via WebRTC. After 40s in this environment the recording of this feed pops up suddenly and replaces the video-feed with the actor on the left side. The users should realise now that reality is not what it seems in this video-chat. It is fake, even time seems to be unpredictable. After 5s looking at their own recorded feed, a popup on top of the screen asks the user if she wants to feed this recording of her into a next round to confuse other people. The question here is why a user would do that. In the user testing most users wanted to participate in the setup in the next round. As the users dynamically replace the videos on the left and right this could be a self-feeding chat that is never real time - you are always talking to strangers from another time. But for a few seconds they exist for you in real-time with you in this world - until you realize that this was not true. At least according to our concept of time and being. 

As mentioned before we used three.js as the main framework. On top of that we used webRCT extensively, especially the recording and download function of webcam feeds. On the backend python helped us to move around the recorded and downloaded files dynamically from download-location on the local machine to a JS-accessible folder. Python as well helped us to keep track of the position of the videos (left or right) when the browser window gets re-loaded between different users. This was a hack, a node - server would have probably been better for this task - but python was simply quicker. 

We did not use the app in a live setup as we felt we need to refine the code and as well experiment further with porting it to VR. 

So far it was very rewarding for me as I could explore a lot of three.js while working with Chian on the project. WebRTC proofed again to be a great way to set up a life video chat - with a little help from python on the backend it worked as a prototype. The VR version will probably have to run exclusively in unity. This is mainly C# - a new adventure for the next semesters! 


Here a video walkthrough:


On the code side we used a github-repo to backup our files. Here you can find all files including a short code documentation as readme. 





Users turn a giant knob on the 4th floor of ITP and control the brightness of a virtual LED in AR positioned on a physical column in the Tisch Lounge. 


Inside the knob will be a rotary encoder to track the exact position of the element. These values are read and continuously sent to a remote AWS server from a nodeMCU 8266 wifi module inside the knob via serial/python/websockets. On the server they are stored in a txt-file and read into a publicly accessible html-file serving the AR-LED in an AR version of three.js. We will have to test how much of a delay we will have using this communication. A public server seems to be the only way for the audience to access the AR site from outside the local NYU network. The delay might still be acceptable as knob and AR-column are situated on different floors. With iOS 11, AR on mobile is possible now on all platforms using getUserMedia/camera access. 

Here a quick test with the github-example library, a target displayed on a screen and an iPhone running iOS 11:




We found a 3ft long rotating steel arm with bearings and attachment ring in the shop junk-shelf. We will use it as the core rotating element inside the knob. 


This rotating element will be mounted on a combination of wooden (plywood) bases that will stabilize the knob when it is getting moved. The weight of the knob will rest on wheels that are running on a rail/channel in the wide base that features a wave structure inside so that the 'click' of the knob feels more accurate. Inside the knob we will build a structure with 2 solid wooden rings that are the diameter of the knob and are attached to the rotating element. On the outside we will cover the knob with lightweight wooden planks or cardboard tubes. 



We worked on the wooden base for the knob/metal pole using multi-layered plywood to keep the rotary encoder within the same wooden element as the pole - this prevents a damage of the electronics/mechanics once there is a push or tilt towards the sides of the knob.



In collaboration with Brandon Newberg & Anthony Bui.


Screenshot from 2017-11-20 11-13-35.png




  • Tue. 21 Nov. 2017: tech communication finished, fabrication parts ordered

work on automation + fabrication (knob), work on AR script

  • Tue. 28 Nov. 2017: all communications work, knob basic fabrication finished

work on fabrication finish, column, code cleaning, documentation, 

  • Tue. 5 Dec. 2017: final presentation



As in PCOMP I switched from flying solo for my project to a collaboration with Chian Huang: she showed me her project and was open to the idea about working together. Her concept was based on a web-based VR/3D experience with camera inputs including sound. I quickly agreed to the collaboration and after a brainstorming session we altered the initial idea and came up with a rough outline for our project: "Watch yourself watching others, watch others watching you - maybe."


social networks, surveillance, peer pressure


The system should create a VR network structure where multiple agents can interact with each other  - all in google cardboard. As the latter technology will obscure the eyes, the participants are not sure whether they are being watched by the other participants. Audio will be disabled and body parts exchanged between the video feeds. The physical interchangeability of the participants highlights a distinct feature of new social networks: The user is just a datacloud, which can be sold off in parts to the highest bidders. Our obsession with watching ourselves and others is the fundamental interaction of virtual social networks. But we never really know what other network members see. 

Chian's initial sketch is based on a p5 example and augmented with music. 

We based our concept on those video-streams and used webRTC to have multiple participants in a virtual space. All of the sketches are running on a remote server in the cloud. 


After experimenting with getting an external webRTC stream as a texture into p5/WebGL, we decided to switch to three.js  - it seems to offer better rendering and more flexibility with webRTC. 

Here a few three.js example sketches that we might use as starting points for our project:

Screen Shot 2017-11-15 at 6.02.15 AM.png

We still have not finally decided on the aesthetics of this "surveillance" - experience. We would like to keep it open until we both feel more comfortable with three.js  - which seems to be considerably more challenging than p5, but as well a lot more powerful when it comes to 3D rendering. 



laser cut tablet of neural network generated cuneiform

laser cut tablet of neural network generated cuneiform

glossolalia setup.png


glossolalia / speaking in tongues


Are neural networks our new gods?


My installation piece aims to create a techno-spiritual experience with a neural network trained on ancient cuneiform writing for an exhibition audience in an art context. 


The title 'glossolalia / speaking in tongues' of the piece refers to a phenomenon where people seem to speak in a language unknown to them - mostly in a spiritual context. In the art piece both "speaker" (machine) and "recipient" are unaware of the language of their communication: The unconscious machine is constantly dreaming up new pieces of cuneiform tablets that the audience cannot translate. Two things to mention: First, after 3000 years, one of the oldest forms of human writing (means encoding of thoughts) becomes "alive" again with the help of a neural network. Second, it is difficult to evaluate how accurate the new cuneiform is - only a few scholars can fully decode and translate cuneiform today. 

original cuneiform tablet ( wikimedia )

original cuneiform tablet (wikimedia)

part of original Sumerian cuneiform tablet (paper impression)

part of original Sumerian cuneiform tablet (paper impression)

cuneiform generated by neural network

cuneiform generated by neural network

Observing the machine creating these formerly human artifacts in its "deep dream" is in itself a spiritual experience: It is in a way a time-machine. By picking up on the thoughts of the existing, cuneiform writing corpus, the machine breathes new life into culture at Sumerian times. The moment the machine finished training on about 20 000 tablets (paper impressions) and dreamed up its first new tablet, the past 2000 - 3000 year hiatus became irrelevant - for the neural network old Babylon is the only world that exists. 

In the installation piece, the audience get the opportunity to observe the neural network the moment they kneel down on a small bench and look at the top of an acrylic pyramid. This activates the transmission of generated images from the network in the cloud to the audience that hover as an extruded hologram over the pyramid.

cunei_bstract [Recovered].png
side, above: digital abstractions of generated cuneiform

side, above: digital abstractions of generated cuneiform

The audience can pick up headphones with ambient sounds that intensify the experience (optional).

It is important to mention that the network is not activated by the audience: The audience gets the opportunity to observe its constant and ongoing dream. The network is not a slave of the audience, it is regarded as a form of new entity in itself. When an audience member gets up from the bench, the transmission stops - the spiritual moment is over.



-- "H" for "have", "G" for "get" --

PCOMP parts:

  • raspberry pi or nvidia jetson tx2 -- G
  • if raspberry pi: phat DAC for audio out -- G
  • 8266 wifi module -- G
  • 2 x buttons / switches -- H
  • headphones -- G
  • audio cables -- G
  • screen -- G
  • hdmi cable -- H
  • local wifi-network (phone) -- H

ICM parts: 

  • GPU cloud server trained on 20 000 cuneiform tablets (scientific drawings, monochrome) -- H
  • processing sketch -- H

Fabrication parts:

  • wood for kneeling bench -- G
  • wood for column -- G
  • acrylic for pyramid -- H
  • wooden stick for headphone holder -- G

system diagram




  • Tue. 7 Nov. 2017: playtesting with paper prototype / user feedback

work on server-local comm, processing sketch, order/buy fab parts

  • Tue. 14 Nov. 2017: server-local comm works, all fabrication parts received

work on input-local comm, processing sketch, fabrication (peppers ghost)

  • Tue. 21 Nov. 2017: processing sketch finished, input-local comm works

work on automation, fabrication (bench, column)

  • Tue. 28 Nov. 2017: all comms work, basic fabrication finished

work on fabrication finish, code cleaning, documentation, 

  • Tue. 5 Dec. 2017: final presentation






This week I tried using  a javascript emotion-tracking library - generally with mixed results (not accurate and slow, maybe a processor issue or the camera resolution too low) in combination with a bonsai growing sketch (open in Safari, Chrome does not render this sketch properly) I worked on for the past couple of weeks. The idea was to trigger the growth of the digital bonsai not only with the amount of mouse-clicked water but with smiles. 

As the results for the facial-expressions tracking were mixed I decided to update my coffee ground -tarot interface with the possibility for the user to take a snapshot from the webcam stream and use the average brightness of this snapshot for "predicting" the horoscope via the API.

I redesigned the arrangement of the elements to make it better accessible from multiple devices but struggled to get it running on iOS 11 Safari - it asked for access to the camera but then only showed a black frame. I would like to fix this, as the direct camera access only makes sense from a mobile device - no user will use a laptop camera for taking images of coffee ground, the mobile camera instead makes more sense.