Incomplete Reporting: Addressing the Problem of Outcome-Reporting Bias in Educational Research

Outcome-reporting bias is a problem that pervades many research disciplines including education. Outcome-reporting bias involves any time the outcomes presented in a published journal article do not accurately reflect all of the outcomes that were collected throughout the course of a study. In other words, when outcome-reporting bias is present, the information which is disseminated to the academic research community is incomplete and can lead to serious problems over time. Some of these problems include school districts implementing interventions based on incomplete or inaccurate data, as well as the problem that naturally arises for those implementing meta-analysis or systematic review research strategies. Many times these important research methodologies rely on compiling and synthesizing all outcomes that have been collected across a number of different studies. If not all outcomes were made available to the public, the conclusions derived from these important types of studies would be invalid and/or based on potentially biased data. In order to ensure that educators and educational researchers alike are implementing truly effective interventions and that the conclusions derived from meta-analyses and systematic reviews are valid, exploration into factors associated with outcome-reporting becomes necessary. The purpose of this dissertation project was to explore the nature of outcome-reporting bias by identifying potential factors associated with it. This task was achieved by first exploring instances in which outcome-reporting bias was documented in other academic research fields and determining what factors were associated with it in those particular cases. The next step became thoroughly reviewing educational research articles and determining which of those factors appeared to be associated with outcome-reporting bias in educational research. The hope is that by identifying these factors, researchers and journal publishers can work to eliminate them from practice, decreasing the prevalence of outcome-reporting bias in educational research. This study found that not only is outcome-reporting bias present in educational literature, but also that outcomes failing to show statistical significance were 30% more likely to get suppressed than statistically significant outcomes, and outcomes that were not consistent with the publishing author’s original hypothesis were 41% more likely to get suppressed than those that were. In addition, this study also found that researchers holding both faculty and non-faculty positions appeared to exhibit outcome-reporting bias with greater frequency when it comes to statistical significance. Non-statistically significant outcomes were 26% more likely to get suppressed than statistically significant outcomes among individuals holding faculty positions and 50% more likely among non-faculty researchers. These results show that authors have a tendency to withhold outcomes that were not statistically significant as well as those that were not consistent with their initial hypotheses. In addition, there must be some type of pressure present that leads to researchers withholding non-statistically significant outcomes from publication. Further, this study also found that data collected regarding certain populations are more subject to outcome-reporting bias when it comes to statistical significance. When samples are predominantly white, non-statistically significant outcomes are 24% more likely to be suppressed when compared to 73% among predominantly non-white samples. Also, non-statistically significant outcomes are 25% more likely to be withheld among high school samples and 32% more likely to be withheld among non-high school samples