New approaches to Q-matrix validation and estimation for cognitive diagnosis models

A primary purpose of cognitive diagnosis models (CDMs) is to classify examinees based on their attribute patterns. The Q-matrix (Tatsuoka, 1985), a common component of all CDMs, specifies the relationship between the set of required dichotomous attributes and the test items. Since a Q-matrix is often developed by content-knowledge experts and can be influenced by their judgment (de la Torre & Chiu, 2016), this can lead to misspecifications in the Q-matrix that can have unintended consequences on examinees’ classifications. Incorrect classification of examinees can have tremendous impact since some assessments are high-stake and are used to make important decisions about students, such as selection and placement. Previous research based on the Trends in International Math and Science Study (TIMSS) has predominantly focused on comparing the performances of participating countries using their average scores. This study focused on fitting data from the TIMSS with a CDM to obtain estimated attribute profiles that will provide information about skill proficiency of students in the participating countries. However, since the test is not specifically designed for use with a CDM, a provisional Q-matrix was developed with input from content experts. As a preliminary analysis, the TIMSS data was first fitted with the generalized deterministic inputs, noisy, “and” gate (G-DINA) model to obtain examinees’ estimated attribute profiles. An evaluation of the estimated attribute profiles however indicated that there are inconsistencies in classification, which may be due to misspecification in the provisional Q-matrix. To ensure that the provisional Q-matrix is appropriately developed, this dissertation proposes one Q-matrix validation method that can be used to correct possible misspecifications in a Q-matrix, and one Q-matrix estimation method for estimating a Q-matrix from scratch. The proposed methods both integrate the Q-matrix validation procedure (Chiu, 2013) that is based on a nonparametric classification method. The first method, the integrated Q-matrix validation (IQV) technique, uses a joint maximum likelihood estimation (JMLE) procedure for diagnostic classification models (Chiu, Köhn, Zheng, and Henson, 2016) to determine examinees’ attribute profiles that are then integrated into the algorithm of Chiu’s Q-matrix validation method to validate the Q-matrix. In the second method, the two-step Q-matrix estimation (TSQE) method, factor analysis is first applied to the correlation matrix to obtain a provisional Q-matrix. The provisional Q-matrix is then incorporated into the algorithm of Chiu’s Q-matrix validation method, to obtain the true Q-matrix. The viability of both methods was investigated using simulation studies with various conditions. The TIMSS data was re-analyzed with the G-DINA model using modified Q-matrices obtained from analysis with the proposed methods. An evaluation of the updated estimated attribute profiles indicated that some of the inconsistencies in classification that were previously identified have been resolved.Ph.D.Includes bibliographical reference