Whose consensus is it anyway? Scientific versus legalistic conceptions of validity

Paul E. Newton provides an insightful and scholarly overview of central issues in validity theory. As he notes, many of the conceptual problems in validity theory derive from the fact that the word validity has two meanings. First, it indicates whether a test measures what it purports to measure. This is a factual claim about the psychometric properties of a proposed measurement instrument. Many people—including psychometricians and test theorists, as Newton documents—still think that this is the accepted definition in validity theory, which is not the case. In the current consensus definition, the term validity indicates to what extent an interpretation of a test score is justifiable (or a variation on that theme). This is not a factual but an evaluative claim about a given interpretation of the test score, which need not involve measurement at all (Borsboom, Cramer, Kievit, Zand Scholten, & Franic, 2009). As Newton notes, however, validity is so intertwined with measurement that many validity theorists have difficulty maintaining their own position consistently, and often unwittingly slip back into the traditional definition of validity. Newton attempts to resolve this tension by bringing the reference to measurement back into the definition of validity, while maintaining the idea that validity is grounded in test score interpretations. He does so by requiring that the argument for a particular kind of test interpretation be sufficiently strong; namely, measurement interpretations that are entailed by the decisions based on the test score