Inversion from Audiovisual Speech to Articulatory Information by Exploiting Multimodal Data

We present an inversion framework to identify speech production properties from audiovisual in-formation. Our system is built on a multimodal articulatory dataset comprising ultrasound, X-ray, magnetic resonance images, electromagnetic artic-ulography data as well as audio and stereovisual recordings of the speaker. Visual information is captured via stereovision while the vocal tract state is represented by a properly trained articulatory model. The audiovisual-to-articulation relationship is approximated by an adaptive piecewise linear mapping. The presented system can recover the hid-den vocal tract shapes and may serve as a basis for a more widely applicable inversion setup.