Towards Augmented Exploratory Testing

Software systems have an increasing presence in our society. With our infrastructure, such as food and water supply, controlled by complex software systems, it is essential to keep failures to a minimum. Exploratory testing has shown to be a good method of finding bugs that require more complex interactions, as well as to gain insights into how the system under test behaves. However, tool support in this area is lacking. Supporting exploratory testing with automation tools has the potential of freeing humans to spend their time where it is of the highest yield and also aid in covering the vast state space of testing a complex software system in a meaningful way. To be able to engineer such tools, targeting interaction models of contemporary systems, deeper knowledge is needed of the possibilities and limitations of applying available software testing methods. In this thesis we propose and evaluate several approaches that can be used to automatically perform black-box system-level testing, potentially augmenting humans in exploratory testing. We base these approaches on random-based methods, such as property-based testing, and present the results of evaluations on several industry-grade systems. Our results show that the proposed and evaluated methods automatically can find faults in real-world software systems. In addition to fault finding, our methods can also find insights such as deviations between the specification of the system under test and the actual behavior. The work in this thesis is a first step towards augmenting developers and testers in exploratory testing