Approximating probability distributions with short vectors, via information theoretic distance measures

Given a probability distribution p = (p1., pn) and an integer m < n, what is the probability distribution q = (q1., qm) that is 'the closest' to p, that is, that best approximates p? It is clear that the answer depends on the function one chooses to evaluate the goodness of the approximation. In this paper we provide a general criterion to approximate p with a shorter vector q by using ideas from majorization theory. We evaluate the goodness of our approximation by means of a variety of information theoretic distance measures