Direct Approximation of Divergences between Probability Distributions

Approximating a divergence between two probability distributions from their sam-ples is a fundamental challenge in the statistics, information theory, and machine learning communities, because a divergence estimator can be used for various pur-poses such as two-sample homogeneity testing, change-point detection, and class-balance estimation. Furthermore, an approximator of a divergence between the joint distribution and the product of marginals can be used for independence test-ing, which has a wide range of applications including feature selection and extrac-tion, clustering, object matching, independent component analysis, and causality learning. In this article, we review recent advances in direct divergence approxima-tion that follow the general inference principle advocated by Vladimir Vapnik|one should not solve a more general problem as an intermediate step. More speci-cally, direct divergence approximation avoids separately estimating two probability distributions when approximating a divergence. We cover direct approximators of the Kullback-Leibler (KL) divergence, the Pearson (PE) divergence, the relative PE (rPE) divergence, and the L2-distance. Despite the overwhelming popularity of the KL divergence, we argue that the latter approximators are more useful in prac-tice due to their computational efficiency, high numerical stability, and superior robustness against outliers.