Replicability is not reproducibility: Nor is it good science

At various machine learning conferences, at various times, there have been discussions arising from the inability to replicate the experimental results published in a paper. There seems to be a wide spread view that we need to do something to address this prob-lem, as it is essential to the advancement of our field. The most compelling argument would seem to be that reproducibility of ex-perimental results is the hallmark of science. Therefore, given that most of us regard ma-chine learning as a scientific discipline, being able to replicate experiments is paramount. I want to challenge this view by separating the notion of reproducibility, a generally de-sirable property, from replicability, its poor cousin. I claim there are important differ-ences between the two. Reproducibility re-quires changes; replicability avoids them. Al-though reproducibility is desirable, I contend that the impoverished version, replicability, is one not worth having. 1