Toward an Emotionally Intelligent Piano: Real-Time Emotion Detection and Performer Feedback via Kinesthetic Sensing in Piano Performance

A system is presented for detecting common gestures, musical intentions andemotions of pianists in real-time using only kinesthetic data retrieved bywireless motion sensors. The algorithm can detect common Western musicalstructures such as chords, arpeggios, scales, and trills as well as musicallyintended emotions: cheerful, mournful, vigorous, dreamy, lyrical, and humorouscompletely and solely based on low-sample-rate motion sensor data. Thealgorithm can be trained per performer in real-time or can work based onprevious training sets. The system maps the emotions to a color set andpresents them as a flowing emotional spectrum on the background of a pianoroll. This acts as a feedback mechanism for emotional expression as well as aninteractive display of the music. The system was trained and tested on a numberof pianists and it classified structures and emotions with promising results ofup to 92\% accuracy