Discriminative Features via Generalized Eigenvectors

Representing examples in a way that is compati-ble with the underlying classifier can greatly en-hance the performance of a learning system. In this paper we investigate scalable techniques for inducing discriminative features by taking ad-vantage of simple second order structure in the data. We focus on multiclass classification and show that features extracted from the generalized eigenvectors of the class conditional second mo-ments lead to classifiers with excellent empirical performance. Moreover, these features have at-tractive theoretical properties, such as inducing representations that are invariant to linear trans-formations of the input. We evaluate classifiers built from these features on three different tasks, obtaining state of the art results. 1