Application of Search-based Software Testing toNon-functional system properties : A Validated Framework

Context: The importance of testing non-functional properties of the system is growing steadily. Complexity factor of the software is growing proportionally with the growing demands and hence attributes like performance, energy consumption and reliability are proving to be very crucial. Optimizing the software with respect to these properties simultaneously with the functional properties has been found to be a challenge. Search-based Software testing automates this process by using different meta-heuristic techniques. It assures the generation of large number of test cases at a minimal cost. Carrying out testing in this context requires lot of expertise and the aid of a highly flexible approach. There is a strong need of a guide that helps the practitioners(testers) and researchers optimize the non-functional properties using Search-based software testing. Objectives: The objective of the work presented in this thesis is to, first, investigate the non-functional properties, challenges encountered and approaches/suggestions by the practitioners on the application of Search-based software testing in academia and industry. Second objective is to map all the information into a conceptual/ theoretical framework that could be used by Search-based software testing practitioners for optimizing the non-functional system properties. Methods: A qualitative approach has been employed for this thesis work. A literature review with snowball sampling as the search approach was conducted to collect the information regarding the different kinds of systems being tested, the non-functional system properties being optimized, challenges encountered and the tools used for this purpose. Semi-structured interviews are conducted as a part of the validation process and generalizing the results obtained. A total of 9 interviews were conducted. Thematic analysis technique has been used to analyze the collected data. Results: As a result of conducting this research, different dimensions forming the framework have been investigated. The overall result of this study is the formulation fo a framework and that has been validated by conducting interviews. The framework consists of 16 challenges related to the field of Non-functional Search-based software testing. Conclusions: It is found out that Search-based testing for non-functional properties has not been extensively applied in the industry. It has been suggested, used and applied in academia for the most part. Several factors influence the selection of non-functional properties for optimization. Most of the challenges being faced in this subject are inclined towards three areas in Search-based testing. Performance, execution time and energy consumption are three most popularly tested attributes. Further research could be done wherein the framework generated could be put to use by different practitioners and researchers to find out interesting things