Assessing effectiveness and mechanisms in treatment research: A critical look at assay sensitivity and the use of mediation analysis

The author examines two questions: (1) Should “assay sensitivity” be used to determine the appropriateness of a randomized control trial (RCT) for meta-analysis, and (2) How well are published mediation analyses conducted? The first question is relevant to assessing the effectiveness of psychiatric treatments. The second question, though not focusing explicitly on the psychiatric treatment literature, nonetheless has implications for researchers who wish to explore potential mechanisms of treatment. To answer the first question, expected drug vs. placebo effect sizes from (hypothetical) trials with assay sensitivity were calculated over a range of distributional assumptions for combinations of high quality and flawed studies. The use of assay sensitivity to judge RCT quality can lead to substantial bias; this method should not be used. To answer the second question, the author evaluated 50 published mediation analyses with regard to their presentation of three issues (a) testing of the indirect effect, (b) reporting coefficients, and (c) addressing temporal issues. About half of the articles failed to meet minimum criteria for each of these three issues, respectively. Researchers heeding the call of Kraemer et al. (2002, p. 877) to “routinely include and report [mediation] analyses” in RCTs of psychiatric treatment should attend to the recommendations made in this paper. If researchers fail to avoid the problems discussed in this paper, they may end up inadvertently exposing patients to less than ideal treatments, either by not knowing which treatment works best, or by redesigning treatments in a less than ideal manner. If the conclusions in this paper are taken seriously, treatment research will have a better chance of correctly determining the relative effectiveness of treatments, and correctly identifying treatment mechanisms