Dynamic evaluation of the development process of knowledge-based information systems

Codifying expert domain knowledge is a difficult and expensive task. To evaluate the quality of the outcome, often the same domain expert or a colleague of similar expertise is relied on to undertake a direct evaluation of the knowledge-based system or indirectly by preparing appropriate test data. During an incremental knowledge acquisition process, a data stream is available, and the knowledge base is observed and amended by an expert each time it produces an error. Using the kept record of the system's performance, we propose an evaluation process to estimate its effectiveness as it gets evolved. We instantiate this process for an incremental knowledge acquisition methodology, Ripple Down Rules. We estimate the added value in each knowledge base update. Using these values, the decision makers in the organisation employing the knowledge-based information system can apply a cost-benefit analysis of the continuation of the incremental knowledge acquisition process. They can then determine when this process, involving keeping an expert online, should be terminated. As a result, the expert is not kept on-line longer than it is absolutely necessary. Hence, a major expense in deploying the information system-the cost of keeping a domain expert on-line-is reduced. © 2012 Springer-Verlag London Limited