Assessing the impact of programming workshops on biomedical research reproducibility

Objective: Given the growing need for computational reproducibility in the biomedical sciences many libraries have started teaching programming workshops. However, little is known about the extent to which researchers are able to translate their new coding skills into more reproducible workflows. The goal of this study was to assess the impact of programming workshops on the computational reproducibility of biomedical workflows.Methods: This mixed-methods study consisted of semi-structured in-depth interviews with 14 University of California, San Francisco researchers at two points in time: before they participated in a UCSF Library-led introductory R and Python programming workshop, and three months after they completed the workshop. During the interviews the author collected qualitative data on the tools, methods, and processes researchers used in their work, and quantitative data from a questionnaire that measured evidence of computationally reproducible behaviors. The author analyzed the quantitative data to see if there was a statistically significant difference in reproducible behaviors before and after the workshop, and used a thematic analysis approach on the qualitative data to extract the common characteristics of the research workflows before and after, and explore what enabled or prevented researchers from making changes in their workflows.Results: Pre and post scores on a checklist of reproducible behaviors did not change in a statistically significant manner. The qualitative interviews revealed that several participants had made small changes to their workflows including switching to open source programming languages for their data cleaning, analysis, and visualization. Overall many of the participants indicated higher levels of programming literacy, and an interest in further training. Factors that enabled change included supportive environments and an immediate research need, while barriers included collaborators that were resistant to new tools, and a lack of time.Conclusions: While none of the workshop participants completely changed their workflows, many of them did incorporate new practices, tools, or methods that helped make their work more reproducible and transparent to other researchers. This indicates that programming workshops now offered by libraries and other organizations contribute to computational reproducibility training for researchers