Generating Emotionally Relevant Musical Scores for Audio Stories

Highly-produced audio stories often include musical scores that reflect the emotions of the speech. Yet, creating effec-tive musical scores requires deep expertise in sound produc-tion and is time-consuming even for experts. We present a system and algorithm for re-sequencing music tracks to generate emotionally relevant music scores for audio stories. The user provides a speech track and music tracks and our system gathers emotion labels on the speech through hand-labeling, crowdsourcing, and automatic methods. We develop a constraint-based dynamic programming algorithm that uses these emotion labels to generate emotionally relevant musi-cal scores. We demonstrate the effectiveness of our algorithm by generating 20 musical scores for audio stories and show-ing that crowd workers rank their overall quality significantly higher than stories without music