A content analysis of research published in the Journal of Research in Science Teaching from 1985 through 1989

To determine if actual practice was consistent with commonly recommended research methods and procedures, this study examined 130 studies reported over a 5-year period in three volumes of the Journal of Research in Science Teaching (JRST). The results were consistent with similar previous analyses (Shaver & Norton, 1980a, 1980b; Wallen & Fraenkel, 1988a) and indicate that appropriate generalizations beyond the confines of the reported studies may be impossible for most (64%) of the JRST studies surveyed. The findings also show that replication studies, which could be employed to offset deficiencies in generalizability, were not commonly encountered (3%) in these 130 reports. In addition, the study results indicate that many researchers (48%) do not properly restrict their conclusions based on the limits imposed by the accessible populations and samples used; nor do they typically provide possible alternative explanations for the outcomes obtained (76%). These findings prompt the following recommendations: 1. A greater awareness and use of replication as a check on generalizability should be encouraged by the science education community. 2. Clearly defined populations (target and accessible) and fully described samples warrant increased attention as report components from authors, reviewers, and editorial board members of JRST. 3. In light of the difficulties inherent in effecting random selection in educational settings, a greater emphasis should be placed on recognizing the limits that the underlying assumptions of inferential statistics place on research conclusions. The results of this study indicate that the methodological quality of published science education research should remain a concern for both practitioners and readers