An Investigation into Improving Test Effectiveness for Embedded Software

This thesis reports on the investigation of the effectiveness of software testing on embedded systems. The aim was to improve confidence in the current methods employed or to find new methods which could improve the hit rate of defects found before software is sent to a customer. We investigate previous work into software testing effectives and various black box testing methods. There are various Black Box testing methodologies that can be employed to detect errors in systems with varying degrees of success. In this thesis we investigate the transformation of the white box testing technique of Definition Use (DU) Path testing using a RESOLVE like specification, to be applied as black box test method. We do not use RESOLVE it self, instead we defined our own method of automatic test generation based on the principles of RESOLVE. Then we compare this method to more commonly used requirements driven test selection, and pure boundary value analysis (BVA) testing techniques. The results reported in this thesis indicate that BVA and DU test selection methods create tests that are covered by unit and integration tests. The current requirements driven test cases create tests with a combination of features working in tandem. It was found that combination of features was more likely to find defects because developers tests had a lesser focus on this area. The tests generated by the BVA and DU test selection methods did not find any defects that their respective methods were intended to find. This is due to the development team already having tests that covered these areas and defects had been fixed before system tests could be run. Based on the fact that the current test selection methods find defects and the methods we investigated do not, this adds confidence that the system test approach to testing is effective. The investigation of defects found showed that timing related errors are common and that a test selection method designed to find timing related defects would be worth investigating. The investigation also revealed a useful method in automatic generation of test cases. The RESOLVE like specification was used to apply a DU testing as a black box test method. This method showed to be more time efficient at generating test cases than the existing requirements driven approach. Although the test cases did not reveal significant defects, due to the overlap with integration testing, it could be a useful method for developers to generate test cases