Intersemiotic Translation from Film to Audio Description: A cognitive semiotic approach

Audiovisual content is heavily prevalent in many modern societies. Film, in particular, is a favourite form of entertainment to many. Film is by nature both multimodal and polysemiotic, employing the modalities of visual and auditive as well as the semiotic systems of language, gesture, depiction and music. Some audiences do not have immediate access to content using either the auditive or visual modality. Some filmic elements may have to be translated into a different medium for these audiences to be able to experience them. Audio description, the verbal narration of visually depicted on-screen events for audiences with visual impairment, is one such translation possibility (e.g. Braun, 2008; Holsanova, 2016; Rai et al, 2010; Remael et al, 2016). This thesis investigates audio description as an act of intersemiotic translation (Jakobson, 1959; Sonesson, 2014) from a filmic narrative to a narrative constructed using largely spoken language. In other words, it investigates how a multimodal and polysemiotic narrative can be rendered into a unimodal, but still polysemiotic, narrative. An empirical study was undertaken, investigating three different audio descriptions of the same short film from the Visual Into Words project corpus (Matamala & Villegas, 2016). It was predicted that all audio describers would manage to reconstruct the film narrative and that they would do so by transferring original auditive elements of the film such as dialogue, eliminating non-crucial visual content and compensating for key events conveyed visually by translating them into verbal narrations. It was hypothesised that these verbal narrations would reflect a balance between the use of descriptive and narrative language (see Kruger, 2010). Further, it was hypothesised that audio describers would have translated only a subset of visual content into verbal narration and that the three audio describers would vary most in how frequently they translated simple events – that is, actions signified by verb phrases – and vary the least in their frequency of translation of causal relations. The hypotheses were supported with the exception of the hypothesis that predicted that causal relations would have the least variation between audio describers and events would have the most variation. This hypothesis was only partly supported, with spatiality being the category that reflected the most variation between audio describers. Overall, the results indicate that audio describers manage to reconstruct the basic structure of a film narrative but translate less than half of the visually depicted narrative elements. In doing so, they employ translative strategies and use both descriptive and narrative language