Using instructed response times to compare inattentive responding across paper and online modalities: measuring research participant inattention

It is important to examine the equivalence of paper and online data collection methods across several domains. The current study compared paper versus online modalities from a data quality standpoint, with a specific focus on inattentive (i.e., careless) responding by using an easily-implemented method to capture participants’ careless responses: interspersing instructed response items (e.g., “Please select option 3, ‘Strongly disagree,’”) throughout a collection of 15 established measures. A between-subjects design compared the percentage of instructed response items missed (hence, inattentive responses) across three conditions: 1) lab paper, 2) lab online, and 3) non-lab online. The non-lab online condition was predicted to have the most inattentive responses, followed by the lab online condition, and lastly the paper condition. This hypothesis was based on the tendency for distracting and multitasking activities to increase in uncontrolled environments (Liu, 2012; Moisala et al., 2016). Additionally, sustained attention decreases with online displays and people tend to spend less time on in-depth reading (Birkerts, 1996) due to fewer spatial cues necessary for comprehension (Liu, 2012; Mangen et al., 2013). 160 college students participated in the study. The results supported my hypothesis that participants in the non-lab online condition would be most inattentive as the non-lab condition missed significantly more instructed response items than the two lab conditions. However, participants in the lab online condition were no less attentive than those in the lab paper condition. Ultimately, these results indicate online data collection is a viable option in lab studies which require sustained attention and that implementing a simple intervention such as instructed response items is an effective and necessary step in identifying inattention. Online data collection in non-lab environmentsmay be developing in a promising direction, but further research is needed to identify and counter the increased tendency for inattention in those environments