A systemic functional perspective on automated writing evaluation: formative feedback on causal discourse

Making explanations is a very important communicative function in academic literacy; several disciplines including science are dominated by causal explanations (Mohan & Slater, 2004; Slater, 2004; Wellington & Osborne, 2001). For academic success, students need to write about causes and effects well with the help of their instructors, which means that formative assessment of causal discourse is necessary (Slater & Mohan, 2010). However, manual evaluation of causal discourse is time-consuming and impractical for writing instructors. For this reason, automated evaluation of causal discourse, which current automated writing evaluation (AWE) systems cannot perform, is required. Addressing these needs, this dissertation aimed to develop an automated causal discourse evaluation tool (ACDET) and empirically evaluate learners’ causal discourse development with ACDET in academic writing classes. ACDET was developed using three approaches: a functional linguistic approach, a hybrid natural language processing approach combining rule-based and statistical approaches, and a pedagogical approach. The linguistic approach helped identify causal discourse features by analyzing a small corpus of texts about causes and effects of economic events. ACDET detects seven types of causal discourse features and generates formative feedback based on them: causal conjunctions, causal adverbs, causal prepositions, causal verbs, causal adjectives, and causal nouns. The natural language processing approach allowed for assigning part-of-speech tags to sentences and words and creating hand-coded rules for the detection of causal discourse features. The pedagogical approach determined feedback features of ACDET, and it was informed by the theoretical perspectives of the Interaction Hypothesis and Systemic Functional Linguistics and findings of research on causal discourse development. Causal discourse development with ACDET was empirically evaluated through a qualitative study in which four research questions investigated two criteria of computer-assisted language learning evaluation framework: language learning potential (i.e., focus on causal discourse form, interactional modifications, and causal discourse development) and focus on causal meaning. Participants of the study were 32 English as a second language learners who were students in two academic writing classes. Data consisted of pre- and post-tests, ACDET’s text-level feedback reports, cause-and-effect assignment drafts, screen capturing recordings, semi-structured interviews, and questionnaires. The findings indicate language learning potential of ACDET: ACDET drew learners’ attention to causal discourse form and created opportunities for interactional modifications, however, resulted in limited causal discourse development. Findings also reveal that ACDET drew learners’ attention to causal meaning. This study is an important attempt in the field of AWE to analyze meaning in written discourse automatically and provide causal discourse specific feedback. The fact that empirical evaluation of ACDET was based on process-oriented data revealing how students used ACDET in class is noteworthy. The findings of this study have important implications for the refinement of ACDET, the development of AWE systems, and research on causal discourse development