A Hierarchical Probabilistic Model for Novelty Detection in Text

Topic Detection and Tracking (TDT) is a variant of classification in which the classes are not known or fixed in advance. Consider for example an incoming stream of news articles or email messages that are to be classified by topic; new classes must be created as new topics arise. The problem is a challenging one for machine learning. Instances of new topics must be recognized as not belonging to any of the existing classes (detection), and instances of old topics must be correctly classified (tracking)---often with extremely little training data per class. This paper proposes a new approach to TDT based on probabilistic, generative models. Strong statistical techniques are used to address the many challenges: hierarchical shrinkage for sparse data, statistical "garbage collection" for new event detection, clustering in time to separate the different events of a common topic, and deterministic annealing for creating the hierarchy. Preliminary experimental results show promise. Keyword..