Mining for Complex Patterns in Large Databases Using Unexpectedness

We consider the problem of discovering descriptive models of large, multidimensional datasets containing quantitative and qualitative data. We consider desirable properties of such models, namely accuracy, completeness and simplicity. We use information content as a way of quantifying these properties and analyzing how well a model captures the important, information-rich trends in a dataset. We consider the extent to which various well-known data mining methods, including association rules and clustering, produce models with these properties. We propose a new data mining method, based on identifying unexpected data trends, and demonstrate its effectiveness in discovering accurate, complete and simple models. We show how unexpectedness can describe complex patterns, including important relationships that cannot be captured by association rules or clustering. In our method, the information content of the data itself dictates the complexity of the discovered patterns rather than the dis..