Vocalistener2: A singing synthesis system able to mimic a user’s singing in terms of voice timbre changes as well as pitch and dynamics

This paper presents a singing synthesis system, VocaListener2, that can automatically synthesize a singing voice by mimicking the tim-bre changes of a user’s singing voice. The system is an extension of our previous VocaListener system which deals with only pitch and dynamics. Most previous techniques for manipulating voice tim-bre have focused on voice conversion and voice morphing, and they cannot deal with the timbre changes during singing. To develop Vo-caListener2, we constructed a voice timbre space on the basis of var-ious singing voices that are synchronized under pitch, dynamics, and phoneme by using VocaListener. In this space, the timbre changes can be reflected in the synthesized singing voice. The system was evaluated by the Euclidean distance in the space between an esti-mated result and a ground-truth under closed/open conditions. Index Terms — Singing synthesis, Voice timbre changes, Singing information processin