Psychometric Evaluation of Memory and Concentration Sections in the Standardized Assessment of Concussion

The purpose of this study is to evaluate validate the memory and concentration sections of the Standardized Assessment of Concussion (SAC) from the SCAT5. Two studies were conducted. The first study examined item difficulty and discrimination using 10 new word lists of 3 versions in the immediate memory section using item analysis based on classical test theory and to equate scores earned on different versions of 10 word lists using test equating methods. The second evaluated the validity of digits backwards in the concentration section, and evaluated current and new scoring methods for the digit backwards. Two hundred young adults with no previous history of a concussion or head injury in the previous 6 months were tested and a convenient sampling method was used for this study. Participants were tested on orientation, immediate memory, digits backwards, months in reverse order, and delayed memory sequentially. Participants visited a total of 3 times in this study for testing lists A, B, and C (counter-balanced design) of immediate memory, and testing list A and two of lists B-F (anchor-test design) of concentration. In order to minimize a potential learning and/or memory interference effect, 48 hours occurred between each assessment. The first study used item analysis to examine item difficulty and discrimination based on classical test theory on 10 new word-lists of 3 versions in the memory section and to connect scores earned on different versions of 10 word-list using test equating methods. The second study evaluated the validity of digits backwards in the concentration section and evaluated the current and new scoring methods for the digit backwards using item analysis. Results of the first study were as follows: 1) the 3 lists of immediate and delayed memory had most items that obtained acceptable difficulty and discrimination. List A had 20 (100%), list B had 17 (85%), and list C contained 19 (95%) items that met both criteria; 2) to equate lists A and B, list A and C, smoothed equipercentile method and linear method were selected using root mean squared difference (RMDS) and mean signed difference (MSD). Based on the results of smooth equipercentile and linear equating, we developed a raw to raw rounded scores conversion table to equate list A and B / list A and C of immediate memory. Results of the second study were as follows: 1) item determination indicated that 3, and the second row of 4 strings had an unacceptable item. However, 5 and 6 strings obtained acceptable both of item difficulty and discrimination; 2) We found that all 3 strings had unacceptable items using 3 scoring methods. In 4 strings, however, it was found that item determination was improved when using method 2 scoring. Therefore, it might be that method 2 scoring is a better scoring method than traditional and method 1 scoring. Our study demonstrated the 3 versions in the memory sections had appropriate validity by an adequate amount of acceptable items. Furthermore, most items in each version have acceptable psychometric properties, enhancing the validity of baseline SAC score for assessing concussions’ effects. In addition, we provided a scores conversion table of immediate memory for improving the validity of baseline SAC scores. However, the new 6 lists of digits backwards in the concentration section need to be modified to have adequate validity. In addition, it was found that the new scoring method proposed by SAC is different from the scoring method we tested. Therefore, clinicians and practitioners using SAC need to be careful when recognizing and interpreting SAC\u27s problems when using SCAT5