A MEASURE OF RELIABILITY USING QUALITATIVE DATA1

In many research activities, the data is unordered or qualitative. In such circumstances, inter-rater reliability is usually measured by calculating a percentage of agreement score between judges. The pre-sent paper expands on an equation first introduced by Goodman and Kruskal for obtaining a reliability measure of one item. This formula determines inter-rater reliability for a series of items across many sub-jects. The statistic that results is easily interpreted and in many ways is analogous to the conventional reliability for quantitative data. EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT 1975, 35, 843-846. IN many types of research activities, it is necessary to obtain a reliability measure for qualitative or unordered data. The procedures that are presently available cannot handle such data using the classical reliability measures. Finn’s (1970) method assumes interval type data, and Goodman and Kruskal’s (1954) formula for handling reliability of unordered data is good for only one item at a time. This paper expands on the latter’s formulation and discusses an approach for calculating the reliability of a series of items. In this way, the procedure is analogous to the usual reliability determination for an achievement test or an attitude scale