An Experimental Set of Hand Gestures for Expressive Control of Musical Parameters in Realtime

This paper describes the implementation of Time Delay NeuralNetworks (TDNN) to recognize gestures from video images.Video sources are used because they are non-invasive and do notinhibit performer's physical movement or require specialistdevices to be attached to the performer which experience hasshown to be a significant problem that impacts musiciansperformance and can focus musical rehearsals and performancesupon technical rather than musical concerns (Myatt 2003).We describe a set of hand gestures learned by an artificial neuralnetwork to control musical parameters expressively in real time.The set is made up of different types of gestures in order toinvestigate:-aspects of the recognition process-expressive musical control-schemes of parameter mapping-generalization issues for an extended set for musicalcontrolThe learning procedure of the Neural Network is describedwhich is based on variations by affine transformations of imagesequences of the hand gestures.The whole application including the gesture capturing isimplemented in jMax to achieve real time conditions and easyintegration into a musical environment to realize differentmappings and routings of the control stream.The system represents a practice-based research using actualmusic models like compositions and processes of compositionwhich will follow the work described in the paper