Reproducible and replicable comparisons of methods controlling false discoveries in computational biology.

With the advancement of high-throughput technologies, data and computing have become key components of scientific discovery in biology. New computational methods to analyze genomic data are constantly being developed, with several methods often addressing the same biological question. As a result, researchers are now faced with the challenge of deciding between a plethora of tools, each leading to slightly different answers. For several common analyses in computational biology, benchmark comparisons have been published to help users pick an appropriate tool from a subset of alternatives. Despite the popularity of these comparisons, the implementation is often ad hoc, with little consistency across studies. To address this problem, we developed SummarizedBenchmark, an R package and framework for organizing and structuring benchmark comparisons. SummarizedBenchmark defines a general grammar for benchmarking and allows for easier setup and execution of benchmark comparisons, while improving the reproducibility and replicability of such comparisons. Using this framework, we perform a systematic benchmark of several recently developed false discovery rate (FDR)-controlling methods for multiple testing correction. These modern methods have the potential to improve power in biological studies by leveraging additional pieces of information available in the data ("informative covariates") to prioritize, weight, and group hypotheses. We investigate the advantages and limitations of these methods against classical FDR-controlling methods across six biological cases studies and various simulation settings. We provide a summary of our findings as a practical guide to aid users in the choice of methods to correct for false discoveries in future studies.Non UBCUnreviewedAuthor affiliation: Dana-Farber Cancer InstitutePostdoctora