resonance
The purpose of resonance is to place the viewer in direct confrontation with their own emotions. The work emphasizes that the question “PANIC – yes/no?”, which served as the Ars Electronica theme in the year of its presentation, can only be answered personally. Here is some footage from the installation at Ars Electronica 2025:
Data collection
The installation collects biometric data from participants using a Muse 2 EEG headset, a GSR sensor, and facial expression recognition to predict their emotional state. It employs multiple models for emotion recognition, mapping the resulting signals onto the Valence-Arousal-Dominance (VAD) space. The EEG model was trained on the NeuroSense dataset, following the methodology described in this article. The GSR model, trained on the WESAD dataset, classifies only whether the participant is aroused or not, while the facial expression recognition model categorizes input into six discrete emotions: neutral, sadness, happiness, anger, suprise and fear. A web application was used to monitor the signals and classifications, allowing intervention when sensor malfunctions occurred.
A monitoring application was developed in Flask to provide real-time feedback on the participant’s state and the reliability of the sensors. The interface displayed EEG, GSR, and facial expression signals together with their classifications on the Valence–Arousal–Dominance (VAD) space. This allowed me to supervise the installation during use, detecting malfunctioning sensors or inconsistent classifications.
Translating emotions into sound and visuals
The sound design was created using Max MSP alongside a Node server that exposes an API for integration with Max. The project blends various generative composition techniques with hard-coded triggers that activate in response to the emotions detected. The composition is organized into four elements: harmony, melody, percussion, and FX. Harmony and FX were found to be most effective in influencing the valence of the sound, with musical modes correlating to emotional value and FX triggering immediate responses. Melody and percussion, on the other hand, were particularly effective in modulating arousal, with the density of events having the strongest impact.
The visual component was developed in TouchDesigner, combining generative elements with pre-recorded video. Some visuals were created using mathematical models, such as Voronoi diagrams, while others were based on time-lapse footage enhanced with various visual effects. The visuals are synchronized to the rhythm of the sound, enhancing immersion. A key feature of the visual stream is that participants see their own reflection distorted, eliciting a strong emotional response.
