Combinatorial Modelling and Testing of PLC Software using ACTS

Testing is a crucial and necessary part of software development used to detect faults and make sure that the software works properly. One testing method proposed in research is named combinatorial testing, where a test suite covers all combinations of parameter values in a certain way. There is some evidence that indicated that most faults are triggered by a few number of parameter value combinations. There have been various tools proposed for using combinatorial testing in practice. ACTS is one of the most popular combinatorial research tools. Even if ACTS has been used in several industrial projects, few studies have shown how such a combinatorial testing tool can be used for modelling and test generation for industrial control software. In this study we apply modelling and test generation using ACTS on a Programmable Logic Control (PLC) software. This kind of software is used in the safety critical domain for controlling different computer devices, such as a train control management system, which is in charge of many of the critical and safety related procedures on a train. Testing this kind of software is very important because failures can contribute to the loss of lives and money. We show how ACTS can be applied to PLC software. We evaluated ACTS in terms of applicability (i.e., how can ACTS can be applied directly on modelling the PLCs) and efficiency in terms of generation time and test suite size. We used 17 PLC programs provided by Bombardier Transportation. Based on the number of inputs in a program we divided the programs into three complexity categories: small, medium and large programs. An input space model was created for each program, where the needed information was obtained from both the programs and the engineers writing the programs. Each model was created as a system in the graphical interface of ACTS. The different algorithms and combinatorial techniques supported by ACTS were used to generate test suites by varying coverage of the parameter combinations (i.e. t-way) in the command line of ACTS. In this study we used a cut-off time of 1 hour for the test generation, as this is a realistic estimation based on discussion with industrial engineers. Our results showed that not all combinations of algorithms and combinatorial strengths could generate a test suite within the cut-off time. We argue that the results of the the modelling process and the results showing the efficiency of the test generation tool can be useful for practitioners considering to use combinatorial testing for PLC software.