Approaching Individual Differences Questions in Cognitive Control: A Case Study of the AX-CPT

Investigating individual differences in cognition requires addressing questions not often thought about in standard experimental designs, especially those regarding the psychometrics of a task. The purpose of the present study is to use the AX-CPT cognitive control task as a representative case study example to address four concerns that may impact the ability to answer questions related to individual differences. First, the importance of a task\u27s true score variance for evaluating potential failures to replicate predicted individual differences effects is demonstrated. Second, evidence is provided that Internet-based studies (e.g., MTurk) can exhibit comparable, or even higher true score variance than those conducted in the laboratory, suggesting the potential advantages of such data. Third, the need to evaluate and assess psychometrics between theoretically-driven and raw behavioral measures, and how they may show different correlation patterns with an individual difference outcome measure is shown for both internal consistency reliability and test-retest reliability. Finally, the need to restrict generalizations of psychometrics across samples or populations is highlighted by demonstrating differences in true score variance and their consequences in a schizophrenia cohort compared to matched controls