Multi-expert Multi-criteria Decision Making

Expert analysis and decisions are highly valued assets in a wide variety of fields, from social services to grant funding committees. However, the use of experts can be prohibitive due to either lack of availability or cost. As such, it is desirable to be able to replicate such decisions. However, there are many obstacles that impede an accurate simulation of expert decisions. For example, despite looking at the same information, two experts may disagree on the decisions. In addition, a single expert may make inconsistent decisions across similar scenarios. In this work, we focus on multi-criteria decision making and in particular, in the case of multiple experts (ME-MCDM). We examine how multi-criteria decision making techniques can address the multi-experts dimension of the problem, as well as how argumentation networks can inform us about how to aggregate the multiple experts\u27 decisions. Questions that we consider include: (1) How do we determine which expert(s) we should listen to in the event of a disagreement? (2) How do we detect inconsistencies in expert decisions, and (3) How do those inconsistencies impact who we should listen to? We look at experts\u27 decision data in the area of software quality assessment, and we analyze automated decisions that results from using non-discriminatory techniques (tech- niques that take all decisions - even conflicting -into account with the same importance). We reconsider these data, explore the use of argumentation networks, and reflect on the relevance of such approach. We report the results of our preliminary observations and we propose directions for future work