Exploring hierarchical user feedback in email clustering

Organizing data into hierarchies is natural for humans. However, there is little work in machine learning that ex-plores human-machine mixed-initiative approaches to orga-nizing data into hierarchical clusters. In this paper we con-sider mixed-initiative clustering of a user's email, in which the machine produces (initial and re-trained) hierarchical clusterings of email, and the user iteratively reviews and ed-its the hierarchical clustering, providing constraints on the next iteration of clustering. Key challenges include (a) de-termining types of feedback that users will find natural to provide, (b) developing hierarchical clustering and retrain-ing algorithms capable of accepting these types of user feed-back, (c) determining the correspondence between two hier-archical structures, and (d) understanding how user behavior changes during a single feedback session and designing ma-chine strategies that change with the user. Preliminary ex-perimental results of two cases shows that under ideal con-ditions, this mixed-initiative approach requires only 6 min-utes of user effort to achieve email clusterings comparable to those requiring 13 to 15 minutes of manual editing ef-forts