MultiML: A general purpose representation language for multimodal human utterances

We present MultiML, a markup language for the annotation of multimodal human utterances. MultiML is able to rep-resent input from several modalities, as well as the relation-ships between these modalities. Since MultiML separates general parts of representation from more context-specific aspects, it can easily be adapted for use in a wide range of contexts. This paper demonstrates how speech and ges-tures are described with MultiML, showing the principles— including hierarchy and underspecification—that ensure the quality and extensibility of MultiML. As a proof of concept, we show how MultiML is used to annotate a sample human-robot interaction in the domain of a multimodal joint-action scenario