The role of content-rich visuals in the L2 academic listening assessment construct

Despite the growing recognition that second language (L2) listening is a skill incorporating the ability to process visual information along with the auditory stimulus, standardized L2 listening assessments have been predominantly operationalizing this language skill as visual-free (Buck, 2001; Kang, Gutierrez Arvizu, Chaipuapae, & Lesnov, 2016). This study has attempted to clarify the nature of the L2 academic listening assessment construct regarding the role of visual information. This goal was achieved by developing an interpretive argument for including video-based visuals in L2 academic listening tests. Particular attention was paid to the role of content-related visuals that provided graphical illustration, description, or explanation of the auditory listening message. Using Kane’s validity framework, the explanation inference was of primary concern to this study because it is used to justify the measured construct (Kane, 1992; 2004; 2006; 2013). The explanation inference was supported by two types of evidence. First, the performances of 143 English as a second language (ESL) and English as a foreign language (EFL) students on an academic English listening comprehension test were quantitatively analyzed for the effect of delivery mode (i.e., audio-only vs video-based) and its relationships with test-takers’ listening proficiency (i.e., lower vs higher), item video-dependence (i.e., whether or not an item was cued by video), item type (i.e., local vs global), and viewing behavior (self-reported on a scale from 1-did not watch the video to 5-watched all of the video). Analyses were based on both classical test theory (i.e., ANOVA and correlations) and item response theory (i.e., Rasch analysis). In the video-based version of the test, content-rich videos were used, defined as videos containing relevant graphical content-related visual cues for 60% of the video length. The findings showed that video-dependent items were easier with videos than without for both lower-level and higher-level test-takers, regardless of item type. Video-independent items were unexpectedly harder with videos in general. In particular, video-independent global items were harder in the video-based mode than in the audio-only mode for lower-level test-takers. Viewing behavior had a weak positive relationship with listening comprehension, regardless of proficiency. Second, stakeholders’ perceptions about using content-rich videos were investigated. Using a questionnaire, the same 143 test-takers provided their perceptions of test difficulty, motivation towards listening, listening authenticity, and whether content-rich videos should be used in high-stakes academic listening tests. The effects of mode and proficiency on these perceptions were examined. Similarly, 310 ESL and EFL teachers provided their opinions about the effects of content-rich videos on listening difficulty, motivation, authenticity, and using content-rich videos in L2 listening tests. The effects of teachers’ background (i.e., professional location, education level, and teaching-related experience) on their perceptions were examined. Test-takers found the video-based mode easier than the audio-only mode; however, their perceptions of motivation, authenticity, and using videos in tests were not affected by mode. Regarding video use perceptions, test-takers were in favor of including content-rich videos in L2 academic listening tests. Teachers were more favorable towards the video-based mode than the audio-only mode in terms of listening difficulty, motivation, authenticity, and using videos in L2 academic listening tests. The study has discussed how these findings supported the interpretive argument for including content-rich video-based visual information into the assessment construct of L2 academic listening comprehension. Challenges revealed by the findings were also addressed, with limitations acknowledged. The study also offered theoretical and practical implications for the field of L2 assessment. As its primary implication, the study recommends that test developers start using content-rich visual information in L2 academic listening tests