Automated Oracle Generation via Denotational Semantics

Software failure detection is typically done by comparing the running behaviors from a software under test (SUT) against its expected behaviors, called test oracles. In this paper, we present a formal approach to specifying test oracles in denotational semantics for systems with structured inputs. The approach introduces formal semantic evaluation rules, based on the denotational semantics methodology, defined on each productive grammar rule. We extend our grammar-based test generator, GENA, with automated test oracle generation. We provide three case studies of software testing: (i) a benchmark of Java programs on arithmetic calculations, (ii) an open source software on license identification, and (ii) selenium-based web testing. Experimental results demonstrate the effectiveness of our approach and illustrate the success of the application on the software testing