SuÆcient Dimensionality Reduction with Irrelevance Statistics

The problem of unsupervised dimensionality reduction of stochastic variables while pre-serving their most relevant characteristics is fundamental for the analysis of complex data. Unfortunately, this problem is ill dened since natural datasets inherently contain al-ternative underlying structures. In this paper we address this problem by extending the re-cently introduced \SuÆcient Dimensionality Reduction " feature extraction method [7], to use \side information " about irrelevant struc-tures in the data. The use of such irrelevance information was recently successfully demon-strated in the context of clustering via the Information Bottleneck method [1]. Here we use this side-information framework to iden-tify continuous features whose measurements are maximally informative for the main data set, but carry as little information as possi-ble on the irrelevance data set. In statistical terms this can be understood as extracting statistics which are maximally suÆcient for the main dataset, while simultaneously max-imally ancillary for the irrelevance dataset. We formulate this problem as a tradeo op-timization problem and describe its analytic and algorithmic solutions. Our method is demonstrated on a synthetic example and on a real world application of face images, show-ing its superiority over other methods such as Oriented Principal Component Analysis.