Diagnostic Taxa as Open Concepts: Metatheoretical and Statistical Questions about Reliability and Construct Validity in the Grand Strategy of Nosological Revision

Since I find it hard to conceive that a rational mind could think otherwise, I presuppose that, ceteris paribus, careful delineation of the signs, symptoms, and course of a disorder (I cannot interest myself much in the semantic hassle over whether to call it “disease”) so as to increase the reliability of classifying clients or patients is desirable. While reliability and validity are not the same thing, it is a psychometric truism that the former bounds the latter, although it is worth mentioning that the bound is the square root of the reliability, so validity can theoretically be larger. Usually the operative validity (net attenuated construct validity) runs far below that upper bound set by the square root of the reliability coefficient. Hence, alterations in the format of assessment or in the content sampled, which might under some circumstances reduce reliability, could nevertheless increase the net attenuated construct validity. Similarly, changes in content or format that increase reliability may theoretically decrease validity. For instance, an alteration in the open-ended, unstructured format of Rorschach administration (as was attempted during World War II to make it possible to test large numbers of individuals and score reliably without inquiry) seemed to eliminate whatever slight validity the instrument had as usually administered. There is no mystery about this, although it is paradoxical at first look. We may be concerned about the reliability of behavior sampling by two different samplers (“interjudge agreement”) or with the trustworthiness of a sample as drawn by an individual judge (how many marbles do we draw from the urn, and how do we draw them?). In either case, the point is this: Whether an interview behavior or a psychological test item is viewed primar