Towards More Effective Performance Testing

Software on modern computer systems is ubiquitous in our daily lives, running on a wide range of devices, from smart watches to super computers. Such software systems grant us with the versatility of their functionality, and become increasingly complex during years of development to adapt the increasing demands of users. However, the software systems may not be dependable per se, on which we rely more and more in recent years, as the human mistakes in the code could turn to bugs, defects or vulnerabilities. To minimize both the number and the severity of such human mistakes, developers usually test the software before releasing it to public. Software tests verify if the behavior of the underlying software, or software under test (SUT), deviates from a specification, which states the required functional or non-functional properties of the SUT. One of the key aspects of non-functional properties is the software performance, which measures how fast a SUT processes its inputs. In comparison with functional testing, performance testing lacks testing oracles, is flaky with testing environment. In this thesis, a piece of inefficient code causing the SUT to deviate from specification is considered as a performance bug. Developers usually utilize performance testing similar to aforementioned functional testing to identify performance bugs. This thesis aims to improve performance testing by performance mutation testing (PMT) and performance fuzzing. A natural question is, as developers err in the SUT, a test writer may also make mistakes when devising the test suite, and it remains unknown whether performance testing is caliberated correctly to be able to find performance bugs. So, an approach to grade the quality of performance testing is needed. As performance testing usually lacks testing oracles, the first step towards evaluating performance testing is to study and understand existing performance bugs. A complete study of more than 700 performance bugs with the discussion on their semantic commonalities is hence presented. Based on investigated performance bugs as well as the extracted semantics, the mutation testing (MT) technique is adopted to grade the effectiveness of performance testing. MT injects known bugs into SUT, and checks if the testing suite is capable of identifying artificially injected bugs. In addition, MT could also be used to determine if injected bugs are covered, thus determining the code coverage of the testing suite. Performance bugs to be injected are synthesized based on the fault models derived from the aforementioned investigation. In this thesis, an effective PMT framework is developed and evaluated. PMT tells merely how effective performance testing is, and a lot of manual efforts are nevertheless needed to find proper inputs for a test case. In recent years, fuzz testing (or fuzzing for short), was utilized to search for proper inputs for SUT automatically. Despite existing approaches exploited the capability of fuzzing to identify performance bugs, the existing discussion mainly focuses on how a performance fuzzer could explore the longest execution path. An early experiment shows that each node of the execution path, on which fuzzing is guided, may not yield the same performance impacts. As many factors could impact the performance of generated inputs, e.g., fuzzing timeout limits, this thesis reevaluates a performance testing framework, known as PERFFUZZ, to provide general recommendations on using performance fuzzing. In summary, this thesis a) provides a dataset of performance bugs, and b) presents a PMT framework accepting performance bug semantics, and c) investigates significant configurations of performance fuzzing and provides suggestions on the effective usage of performance fuzzing