Assertion-driven development: Assessing the quality of contracts using meta-mutations

Abstract—Agile development methods have gained momentum in the last few years and, as a consequence, test-driven development has become more prevalent in practice. However, test cases are not sufficient for producing dependable software and we rather advocate approaches that emphasize the use of assertions or contracts over that of test cases. Yet, writing self-checks in code has been shown to be difficult and is itself prone to errors. A standard technique to specify runtime properties is design-by-contract (DbC). But how can one test if the contracts themselves are sensible and sufficient? We propose a measure to quantify the goodness of contracts (or assertions in a broader sense). We introduce meta-mutations at the source code level to simulate common programmer errors that the self-checks are supposed to detect. We then use random mutation testing to determine a lower and upper bound on the detectable mutations and compare these bounds with the number of mutants detected by the contracts. Contracts are considered “good ” if they detect a certain percentage of the detectable mutations. We have evaluated our tools on Java classes with contracts specified using the Java Modeling Language (JML). We have additionally tested the contract quality of 19 implementations, written independently by students, based on the same specification