Evaluation of Multi-Agent Systems: proposal and validation of a metric plan

In the MAS evaluation research field there are still few works devoted to evaluating systems’ efficacy, and none of these aimed to measure the adequacy of the MAS in terms of rationality, autonomy, reactivity and environment adaptability. A reliable evaluation method should be general enough to estimate the success of the multi-agent paradigm in different domains, measuring the performances of each single agent and then of the entire MAS. Moreover, it should be able to relate these measures to the environment complexity, that embodies the complexity of the problem solved by the MAS. In this paper a method for evaluating static multi-agent systems is presented and its validation described. The main novelties of the method are that it allows the MAS to be evaluated in the context of the environment in which it will operate, and its adequacy to the environment to be judged from the viewpoints of both the designer, wishful to measure the quality of the designed MAS, and the evaluator, wishful to verify the adequacy of several MASs in a specific context. A validation of the method is described, carried out by evaluating two MASs: the GeCo-Automotive system and a Multi-Agent Tourism Recommender system