The development and validation of an automatic-item generation measure of cognitive ability

Cognitive ability is perhaps the most studied individual difference available to researchers, being measured quickly and effectively while demonstrating a predictable influence on many life outcomes. Historically, the evolution of the psychometric study of cognitive abilities has pivoted on the development of new and better methodologies allowing for a more complete and efficient capture of intellect. For instance, recent advances in computer and Internet technology have largely replaced traditional pencil-and-paper methods, allowing for innovative item development and presentation. However, concerns regarding the potential adverse impact and test security of online measures of cognitive ability, particularly in unproctored situations, are well documented and have limited the use of such measures in organizational settings. Methods, such as the use of multiple test forms and computer adaptive testing coupled with item exposure algorithms, have addressed some test-security concerns. However, these methods require the costly and tedious development of extensive item pools. The burgeoning area of automatic item generation potentially addresses many of the test-security and item-development concerns through the creation of assessment items based solely on an item model and a computer algorithm. Moreover, once the elements that contribute to item difficulty are calibrated, the psychometric properties of the items are known, meaning that little to no human review of the items is required before their use. The purpose of the current study was to develop an experimental non-verbal measure of cognitive ability through automatic item generation, using an innovative item type. Using a sample of 333 adults, the results of the current analysis support the proposed cognitive model\u27s ability to explain item difficulty. Likewise, the temporal stability and predictive validity of the experimental measure are supported. In doing so, the experimental measure answers some of the test-security and item-generation concerns that are associated with the development and administration of cognitive-ability measures in organizational settings