An empirical comparison of compiler testing techniques

Compilers, as one of the most important infrastructure of today's digital world, are expected to be trustworthy. Dif-ferent testing techniques are developed for testing compilers automatically. However, it is unknown so far how these test-ing techniques compared to each other in terms of testing effectiveness: how many bugs a testing technique can find within a time limit. In this paper, we conduct a systematic and comprehensive empirical comparison of three compiler testing techniques, namely, Randomized Differential Testing (RDT), a variant of RDT|Different Optimization Levels (DOL), and Equiv-alence Modulo Inputs (EMI). Our results show that DOL is more effective at detecting bugs related to optimization, whereas RDT is more effective at detecting other types of bugs, and the three techniques can complement each other to a certain degree. Furthermore, in order to understand why their effective-ness differs, we investigate three factors that inuence the effectiveness of compiler testing, namely, effciency, strength of test oracles, and effectiveness of generated test programs. The results indicate that all the three factors are statistically significant, and effciency has the most significant impact