Quark: Towards better quality specification miners

Softwares are often built without specification. Tools to automatically extract specification from software are needed and many techniques have been proposed. One type of these specifications – temporal API specification – is of-ten specified in the form of automaton (i.e., FSA/PFSA). There have been many work on mining software temporal specification using dynamic analysis techniques; i.e., anal-ysis of software program traces. Unfortunately, the issues of scalability, robustness and accuracy of these techniques have not been comprehensively addressed. In this paper, we describe a framework that enables assessments of the performance of a specification miner in generating temporal specification of software through traces recorded from its API interaction. Our framework re-quires the temporal specification produced by the miner to be expressed as probabilistic finite state automaton (PFSA). The framework accepts a user-defined simulator PFSA and a specification miner. It produces quality assurance mea-sures on the specification generated by the miner. We in-vestigate metrics used in these measures by adapting tech-niques found in artificial intelligence, program analysis, bioinformatics and data mining to the software specifica-tion domain. Extensive experiments on two specification miners have been performed to evaluate the effectiveness of the proposed quality assurance measures.