Maximum entropy discrimination

We present a general framework for discriminative estimation based on the maximum en-tropy principle and its extensions. All calculations involve distributions over structures and/or parameters rather than specic settings and reduce to relative entropy projections. This holds even when the data is not separable within the chosen parametric class, in the context of anoma-ly detection rather than classication, or when the labels in the training set are uncertain or incomplete. Support vector machines are naturally subsumed under this class and we provide several extensions. We are also able to estimate exactly and eÆciently discriminative distri-butions over tree structures of class-conditional models within this framework. Preliminary experimental results are indicative of the potential in these techniques.