Using Global Complexity Measures to Assess Second Language Proficiency: Comparing CLIL and non-CLIL learners of Dutch and English in French-speaking Belgium

Within the framework of an interdisciplinary project on Content and Language Integrated Learning (CLIL) in French-speaking Belgium , we will present a first step in the analysis of the L2 language proficiency of French-speaking learners of English and Dutch. A comparison will be made between CLIL and non-CLIL secondary school pupils on the basis of L2 written productions. The central focus will be on syntactic and lexical complexity, lexical diversity and readability. Research shows that it is possible to measure learners’ global proficiency by using different types of metrics. Many researchers valued syntactic complexity measures in second language writing in order to assess learners’ proficiency (e.g. Ortega 2003; Lu 2010) or writing quality (e.g. Beers & Nagy 2007) in a target language. Others like Crossley et al. (2011) showed that lexical indices (e.g. word frequency, lexical diversity, etc.) are reliable predictors of the proficiency level of language learners based on their written productions. Even though Bulté & Housen (2015, p.67) warn against the “overall validity of complex measures” when gauging L2 writing development or quality, half of the measures they used succeeded in capturing “growth in writing proficiency”. Psycholinguistic studies demonstrate that language users and learners are highly sensitive to frequency of input at the construction, type and token level (Ellis 2002; Ellis & Cadierno 2009; Goldberg 2010). Since CLIL programs provide more target language input than non-CLIL programs and can therefore be considered closer to L1 acquisition because of their inherent usage-based approach, we can expect a more native-like acquisition of the target language, i.e. a more native-like level of syntactic and lexical complexity in the writing of CLIL pupils. In order to test this hypothesis, we have collected 473 written productions of CLIL and non-CLIL pupils learning Dutch or English as a foreign language – 89 by CLIL English learners, 92 by non-CLIL English learners, 133 by CLIL Dutch learners and 98 by non-CLIL Dutch learners. Each pupil wrote two e-mails (one in French and one in Dutch/English) about similar topics and of similar text length. Using Word Count Tool, we obtained preliminary results which seem to be in line with our hypothesis. We found higher scores for average text length (in English 320,6 vs. 260,3 words; in Dutch 280,2 vs. 195,1 words) and sentence length (in English 12,2 vs. 10,1 characters; in Dutch 10,1 vs. 9,7 characters) among the CLIL learners of English and Dutch. We observed similar average word length and Flesch Reading Ease scores. A deeper analysis will be made using Coh-Metrix (Graesser et al. 2004) in order to extract scores of lexical and syntactic complexity, lexical diversity and readability. Subsequently, we will compare the complexity scores for the CLIL and non-CLIL texts using t-tests and Cramer’s V effect size to measure the strength of these differences. We will also assess the degree of correlation between the complexity metrics using Pearson’s correlation. These analyses will thus allow us to present the impact of CLIL education on a selection of complexity measures in the target Dutch and English written productions. References Beers, S. F. & Nagy, W. E. (2009). Syntactic complexity as a predictor of adolescent writing quality: Which measures? Which genre? Reading and Writing, 22 (2), 185-200. Bulté, B. & Housen, A. (2015). Evaluating short-term changes in L2 complexity development. Círculo de lingüística aplicada a la comunicación, 63, 42-76. Crossley, S. A., Salsbury, T. & McNamara, D. S. (2011). Predicting the proficiency level of language learners using lexical indices. Language Testing, 29 (2), 243-263. Ellis, N. (2002). Frequency effects in language processing: A review with implications for the theories for implicit and explicit language acquisition. Studies in Second Language Acquisition, 24 (2), 143-188. Ellis, N. & Cadierno, T. (2009). Constructing a Second Language. Introduction to the special section. Annual Review of Cognitive Linguistics, 7, 111-139. Goldberg, A. E. (2006). Constructions at Work: The nature of Generalization in Language. Oxford: Oxford University Press. Graesser, A. C., McNamara, D. S., Louwerse, M. M. & Cai, Z. (2004). Coh-Metrix: Analysis of text on cohesion and language. Behavior research methods, instruments & computers, 36 (2), 193-202. Lu, X. (2010). Automatic analysis of syntactic complexity in second language writing. International Journal of Corpus Linguistics, 15 (4), 474-496. Ortega, L. (2003). Syntactic Complexity Measures and their Relationship to L2 Proficiency: A Research Synthesis of College-level L2 Writing. Applied Linguistics, 24 (4), 492-518