Toward Music Communication with Supplemental Visual Information

We recognized that college students who are deaf and hard of hearing are also interested in music and have been working toward creating an environment in which both people with and without hearing impairments can enjoy music together. Visual assistance raises one possibility of enhancing information that music implies. We have developed a system music with motion pictures (MPM), which generates motion pictures from drawings. MPM animates a drawing with sound data, thus providing a motion picture that is synchronized with the sound data. We generated a set of motion pictures using a musical performance and a drawing that were intended to convey the same specific emotion. Then, we conducted an experiment to investigate whether a musical performer's emotional intention was better conveyed to listeners when motion pictures were provided with the music. While the results matched our expectations in that the motion pictures improved the recognition of the emotion in the music, we observed conflict of recognition when combining information from several media and generating motion pictures that yielded a better recognition of emotion