Evaluating the stability and credibility of ontology matching methods

Abstract. Ontology matching is one of the key research topics in Se-mantic Web. In the last few years, many matching methods have been proposed to generate matches between different ontologies either auto-matically or semi-automatically. To select appropriate ones, users need some measures to judge whether a method can achieve the similar com-pliance even on one dataset without reference matches and whether such a method is reliable w.r.t. its output result along with the confidence. However, widely-used traditional measures like precision and recall fail to provide sufficient hints. In this paper, we design two novel evaluation measures to evaluate stability of matching methods and one measure to evaluate credibility of matching confidence values, which help an-swer the above two questions. Additionally, we carry out comparisons among several carefully selected methods systematically using our new measures. Besides, we report some interesting findings such as identifying potential defects of our subjects.