Gemund. Empirical Evaluation of Diagnostic Algorithm Performance Using a Generic Framework

A variety of rule-based, model-based and data-driven techniques have been proposed for detection and isolation of faults in phys-ical systems. However, there have been few efforts to comparatively analyze the perfor-mance of these approaches on the same sys-tem under identical conditions. One reason for this was the lack of a standard frame-work to perform this comparison. In this pa-per we introduce a framework, called DXF, that provides a common language to repre-sent the system description, sensor data and the fault diagnosis results; a run-time archi-tecture to execute the diagnosis algorithms under identical conditions and collect the di-agnosis results; and an evaluation compo-nent that can compute performance metrics from the diagnosis results to compare the al-gorithms. We have used DXF to perform an empirical evaluation of 13 diagnostic algo-rithms on a hardware testbed (ADAPT) at NASA Ames Research Center and on a set of synthetic circuits typically used as bench-marks in the model-based diagnosis commu-nity. Based on these empirical data we ana-lyze the performance of each algorithm and suggest directions for future development. 2/20107/2010 This is an open-access article distributed under the terms of the Creative Commons Attribution 3.0 United States License, which permits unrestricted use, distri-bution, and reproduction in any medium, provided the original author and source are credited.