Text Alignment from Bimodal Mathematical Expression Sources

International audienceIn this paper we propose a new approach tomerge mathematical expression recognition results coming fromhandwriting and speech modalities. Using a bimodal descriptionof mathematical expressions allows taking advantage of thecomplementarities between both signals, and can disambiguatesituations were a single modality would not be clear enough. Tocombine the signals coming from both modalities, we propose torepresent them in the same space as a textual description. First,from the handwriting signal, we generate the Nbest mathematicalexpressions; each of them is next translated as different possiblestrings. From the audio signal, an automatic speech recognitionsystem provides a transcript, which is also available as a string.A string comparison algorithm is achieved to select the bestmathematical expressions. This bimodal system is evaluated onreal bimodal data from the HAMEX dataset and the results arecompared to a single modality (handwriting) based system