Mining Surprising Patterns and Their Explanations in Clinical Data

We present a new technique for interactively mining patterns and generating explanations by harnessing the expertise of domain experts. Key to the approach is the distinction between what is unexpected from the perspective of the computational data mining process and what is surprising to the domain expert and interesting relative to their needs. We demonstrate the potential of the approach for discov-ering patterns and generating rich explanations in a clinical domain. Discovering interesting facts in clinical data is a grand challenge, since medical practitioners and clinicians generally have exceptional knowl-edge in the problem domain in which they work, however this knowl-edge is typically difficult to isolate computationally. To identify the desired surprising patterns, we formally record user knowledge, and use that knowledge to filter and constrain the output from an objective data mining technique, with the user making the final judgement as to whether or not a rule is surprising. Specifically, we introduce an un-expectedness algorithm based on association rule mining and Bayesian Networks and a ε-explanations technique for explanation generation to identify unexpected patterns. An implemented prototype is success-fully demonstrated using a large clinical database recording incidence, prevalance and outcome of dialysis and kidney transplant patients