What Test Oracle Should I Use for Effective GUI Testing

Test designers widely believe that the overall effective-ness and cost of software testing depends largely on the type and number of test cases executed on the software. In this paper we show that the test oracle used during test-ing also contributes significantly to test effectiveness and cost. A test oracle is a mechanism that determines whether a software executed correctly for a test case. We define a test oracle to contain two essential parts: oracle informa-tion that represents expected output, and an oracle proce-dure that compares the oracle information with the actual output. By varying the level of detail of oracle information and changing the oracle procedure, a test designer can cre-ate different types of test oracles. We design 11 types of test oracles and empirically compare them on four software sys-tems. We seed faults in each software to create 100 faulty versions, execute 600 test cases on each version, for all 11 types of oracles. In all, we report results of 660,000 test runs on each software. We show (1) the time and space re-quirements of the oracles, (2) that faults are detected early in the testing process when using detailed oracle informa-tion and complex oracle procedures, although at a higher cost per test case, and (3) that employing expensive oracles results in detecting a large number of faults using relatively smaller number of test cases