Parameterized Unit Testing in the Open Source Wild

With recent advances in test generation research, powerful test generation tools are now at the fingertips of developers in software industry. For example, Microsoft Research Pex, a state-of-the-art tool based on dynamic symbolic execution, has been shipped as IntelliTest in Visual Studio 2015. For test inputs automatically generated by such tool, to supply test oracles (beyond just uncaught runtime exceptions or crashes), developers can write formal specifications such as code contracts in the form of preconditions, postconditions, and class invariants. However, just like writing other types of formal specifications, writing code contracts, especially postconditions, is challenging. In the past decade, parameterized unit testing has emerged as a promising alternative to specify program behaviors under test in the form of unit tests. Developers can write parameterized unit tests (PUTs), unit-test methods with parameters, in contrast to conventional unit tests, without parameters. PUTs have been popularly supported by various unit testing frameworks for .NET along with the recent JUnit framework. However, there exists no study to offer insights on how PUTs are written by developers in either proprietary or open source development practices, posing barriers for various stakeholders to bring PUTs to widely adopted practices in software industry. To fill this gap, in this paper, we present the first empirical study of parameterized unit testing conducted on open source projects. We study hundreds of parameterized unit tests that open source developers wrote for these open source projects. Our study findings provide valuable insights for various stakeholders such as current or prospective PUT writers (e.g., developers), PUT framework designers, test-generation tool vendors, testing researchers, and testing educators.Ope