ON EVALUAmNG AI SYSTEMS FOR MEDICAL DIAGNOSIS

Among the difficulties in evaluating AI-type medical diagnosis systems are: the intermediate conclusions of the AI system need to be looked at in addition to the “final ” answer; the “superhuman human ” fallacy must be resisted; both pro- and anti-computer biases during evaluation must be guarded against; and methods for estimating how the approach will scale upwards to larger domains are needed We propose a type of Turing test for the evaluation problem, designed to provide some protection against the problems listed above We propose to measure both the accuracy of diagnosis and the structure of reasoning, the latter with a view to gauging how well the system will scale up TESTINGSYSTEMS WITHMULTIPLEDIMENSIONS toper-formance and possibly designed to meet multiple objec-tives, especially involving many subjective components, is inherently fraught with pitfalls. Evaluation of AI systems for medical decision-making (AIM systems) is no exception. The problem of evaluation of such systems is very important, however, since these sophisticated approaches should even-tually result in systems with real benefits in clinical practice. Their widespread introduction is not possible until reliabl