Annotating Significant Relations on Multimedia Web Documents

Annotation is a fundamental activity for information extraction. Annotation of large corpora provides a basis for building ontologies and thesauri (Braschler and Schauble 2000), while annotation of existing data helps adding semantics (Volz et al. 2004). The elements to be annotated range from simple text (Kahan and Koivunen 2001), to structured data (Geerts et al. 2006), images (Herve and Boujemaa 2007), and video (Del Bimbo and Bertini 2007), while the annotation content can be as simple as a tag (Zheng et al. 2008a,b) or as complex as a whole new document (Bottoni et al. 2006), most commonly being text. While traditional annotation on paper alter the physical support of the document, digital annotations can be added at will on a digital document without modifying it, still maintaining their relationship to the original document through suitable metadata. Automatic annotation of complex documents is still a hard problem for computational systems, hence any support to the manual work of the human annotators, for example, retrieval of existing information or versatility in managing different types of media is helpful. Moreover multimedia documents can be both the subject and the content of the annotation, making the combined management of documents and annotations a must. On the other hand, as we focus on in this chapter, annotation can prove useful not only for wide communities, but also for groups of interest or at the enterprise level, or even for individuals, allowing them to accumulate knowledge on specific topics by constructing a web of annotations. In this sense, the annotation activity can be used to extract and retrieve two types of information. In the first type, which we may call “objective,” well-known facts, for example, dates, places, or names, are extracted, which need to be related, structured, and documented, possibly according to known, domain-dependent conventions. Ithe second, “subjective,” type, the information which is extracted is related to the user’s goal in the context of a particular activity. Users may want to associate different, possibly unrelated parts of a document, such as salient features of an event or location, or of the psychological profile of a person, or information about a product or service, with a view to what is needed for a given task, for example, a presentation, a lecture, or the construction of a personal archive. Moreover, they want to do so in a nondisruptive way, while perusing the document. The MADCOW system (for Multimedia Annotation of Digital Content Over the Web) (Bottoni et al. 2004, 2006), allows users to annotate web pages containing different types of media with web notes composed of text, images, video, and, in general, any type of digital document, which can then be retrieved from the originally annotated document (or directly through queries to an annotation server), and which can be made public. The MADCOW client is integrated as a bookmarklet on the toolbar of the most common browsers and allows readers to create or retrieve annotations on the current page without interfering with their normal behavior when accessing information on the Web. We have recently enriched MADCOW with the ability to create notes pertaining not only to single blocks of text or to structures within pictures contained in the page, but also referring to any combination of these individual elements. As an example, writers involved in the cooperative construction of a web page, or a scholar reading a scientific document, could create a single note on two portions of text that appear as contradictory, or as a repetition. A detailed analysis of a picture might immediately refer to the text describing it. More sophisticated uses might uncover the logical relations between different parts of the document as described in the first scenario presented in the paper. These actions need not be done by the author of the document, but provide a dynamic construction of the interpretation of the document by its readers, in a sense materializing the notion of “open work” (Eco 1962). To our knowledge, no existing system for manual web annotation tackles the problem of linking a single annotation to different portions of the document. In the following, we introduce our running example on two scenarios, concerning cooperative enrichment of available material between teacher and students, and extraction of structured information by a single user, illustrating the use of MADCOW to organize the two types of information discussed above. We present the notion of multistructure, allowing the creation of multi-notes, which associate a single annotation with several elements of a Web document, and present the relevant data structures and the interaction by which they are created exploiting the MADCOW client. Finally, we show how multi-notes can further be manipulated and reused by readers accessing them