Framework for Statistical Testing of Software Components

The term statistical testing is used when statistics is applied to software testing. Statistics provides sampling theory, a sound theoretical foundation for statistical testing. However, two difficult and important issues must be addressed in statistical testing. The first issue is determining an accurate operational profile. Statistical testing of a software component from a users point of view depends largely on the manner in which the component is used. Characterisation of the population of expected use is referred to as an operational profile. An operational profile is a set of input events and their associated probabilities of occurrence expected in actual operation. The test cases that are executed during a statistical test are a sample from the operational profile. As such, accurate operational profiles are a critical part of statistical testing. Determining an operational profile for the generation of test cases that are representative of actual usage is still a difficult issue despite the considerable research in this area in recent years. Developing an accurate operational profile for software is difficult in general and it is very difficult for many software components because it requires anticipating the future use of the component. The usage behaviour is typically modelled by either Markov models or finite state machines. Previous work has focused on exploring the occurrence of operations and has ignored parameters for operations. This thesis presents a systematic method for deriving operational profiles for software components that uses probabilistic statecharts to model operational profiles and that addresses the issue of parameters for operations. The second issue is development of a test oracle for output evaluation. Test output evaluation is also a difficult and important problem for statistical testing. Most of the potential benefits of testing will be lost if the success or failure of test cases is not assessed. An expected result is needed for each test case to check the test output. The mechanism used for test output evaluation is called a test oracle. A test oracle is an essential part of statistical testing, because a large number of test cases is required to represent the operational usage and the behaviour must be checked for every test case. Test result evaluation using a test oracle is widely acknowledged in the software testing literature as a critical aspect of the testing process. Several methods for developing test oracles, such as those using specifications and documentation, have been reported. Unfortunately development and maintenance of such resources may require considerable effort. A limitation in using such a resource is that the test oracle is only as good as the resource from which it was derived. To address these issues, this thesis presents a technique to develop a test oracle that uses the component to check its own behaviour for test output evaluation. While a number of proposals for statistical testing of software have been made, the execution of test cases and test output evaluation are often not addressed in these proposals. This thesis presents a conceptual framework for statistical testing of software components that supports both test case execution and output evaluation. The framework proposed in this thesis is supported by a prototype tool for test case generation, test case execution and output evaluation. The tool supports a wide range of operational profile approaches for test case generation and a variety of test oracles for output evaluation. This thesis demonstrates the practical viability, flexibility and scalability of the framework and tool support by applying them on several case studies, including a third-party component. Experience with the framework indicates that it can be used successfully for small to medium-sized components, including third-party components such as COTS components. The overall effectiveness of the statistical testing process depends on both the accuracy of the operational profile and the effectiveness of the test oracle to detect and report the errors encountered during testing. This thesis presents an empirical evaluation of different types of operational profiles to determine how accurately the test cases generated from the operational profiles represent the actual usage of the component and different types of test oracles to determine their fault-detection ability