Towards software experts for knowledge discovery:a process model for knowledge consolidation

Abstract The process of knowledge discovery in databases (KDD) is traditionally driven by human experts with in-depth knowledge of the technology used and the domain in which it is being applied. The role of technology in this process is to passively execute the computations specified by the experts, and the role of non-expert humans is limited to a few narrowly defined special cases. However, there are many scenarios where ordinary individuals would benefit from applying KDD to their own personal data, if only they had the means to do so. Meanwhile, KDD experts are looking for more advanced tools and methods capable of coping with the challenges posed by big data. Both these needs would be addressed by autonomous software experts capable of taking on some of the responsibilities of human KDD experts, but there are several obstacles that need to be cleared before the implementation of such experts is feasible. One of these is that while there is a widely accepted process model for kno wledge discovery, there is not one for knowledge consolidation: the process of integrating a KDD result with established domain knowledge. This paper explores the requirements of the knowledge consolidation process and outlines a process model based on how the concept of knowledge is understood in KDD. Furthermore, it evaluates the state of the art and attempts to estimate how far away we are from achieving the necessary technology level to implement at least one major step of the process in software. Finally, the options available for making significant advances in the near future are discussed