EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT VACHA-HAASE ET AL. SAMPLE COMPOSITIONS AND VARIABILITIES IN PUBLISHED STUDIES VERSUS THOSE IN TEST MANUALS: VALIDITY OF SCORE RELIABILITY INDUCTIONS

As measurement specialists, we have done a disservice to both ourselves and our profes-sion by habitually referring to “the reliability of the test, ” or saying that “the test is reli-able. ” This has created a mind-set implying that reliability, once proven, is immutable. More important, practitioners and scholars need not know measurement theories if they may simply rely on the reliability purportedly intrinsic within all uses of established measures. The present study investigated empirically exactly how dissimilar in both composition and variability samples inducting reliability coefficients from prior studies were from the cited prior samples from which coefficients were generalized. The 1994 APA Publication Manual “encouraged ” (p. 18) effect size re-porting in all articles. However, a recent summary of 11 empirical studies of reporting practices in one or two post-1994 volumes of 23 journals indicated that this “encouragement ” has had no or little effect (Vacha-Haase, Nilsson, Reetz, Lance, & Thompson, 2000). In part, the encouragement may have been so ineffective because only encouraging effect size reporting sends a self-canceling mixed message. To present an “encouragement ” in the context of strict absolute standards regarding the esoterics of author note placement, This study was promised in the April response to criticism of EPM editorial policies and the “reliability generalization ” (RG) methods proposed by Vacha-Haase (1998). The senior author may be contacted a