1 The Knowledge Gap Toward Knowledge-Rich Data Mining

Data mining has made tremendous progress in the last ten years. However, a large gap remains between the results a data mining system can provide and taking actions based on them. This gap must be filled by human work, which greatly limits the efficiency of the overall process and the scope of applicability of data mining. For example, data mining can reveal that a purchase of diapers at a supermarket is often accompanied by a purchase of beer, but it cannot hypothesize that the buyer is probably a new father, and propose other products appropriate to this demographic. In general, decision-making involves a chain of inferences, and, while we often have enormous quantities of data relevant to some inference steps, allowing us to automate them by learning from the data, we often have no data at all relevant to other steps, and these become the labor-intensive bottleneck. Clearly, the potential benefits of shrinking this “knowledge gap ” could greatly exceed the benefits obtainable from further improving the parts of the process that are already efficient. Another area where lack of portable, machine-understandable knowledge has a high cost is data preprocessing. It is by now a truism in the data mining community that most of the cost of a mining project is in data gathering, integration, cleaning, transformation, etc.; more generally, in setting up the problem in a way that can be solved by our data mining tools (classification, clustering, etc.). Despite this, progress in this area continues to be sparser than progress in further refining the modeling tools. A large part of this i