The Convex Information Bottleneck Lagrangian

The information bottleneck (IB) problem tackles the issue of obtaining relevant compressedrepresentations T of some random variable X for the task of predicting Y. It is defined as a constrainedoptimization problem that maximizes the information the representation has about the task, I(T;Y) ,while ensuring that a certain level of compression r is achieved (i.e., I(X;T) ≤ r). For practical reasons,the problem is usually solved by maximizing the IB Lagrangian for many values of the Lagrange multiplier. Then, the curve of maximal I(T;Y) for a givenI(X;T) is drawn anda representation with the desired predictability and compression is selected. It is known when Yis a deterministic function of X, the IB curve cannot be explored and another Lagrangian has beenproposed to tackle this problem: the squared IB Lagrangian. In this paper, we (i) present a general family of Lagrangians which allow for the exploration of the IBcurve in all scenarios; (ii) provide the exact one-to-one mapping between the Lagrange multiplierand the desired compression rate r for known IB curve shapes; and (iii) show we can approximatelyobtain a specific compression level with the convex IB Lagrangian for both known and unknown IBcurve shapes. This eliminates the burden of solving the optimization problem for many values of theLagrange multiplier. That is, we prove that we can solve the original constrained problem with asingle optimization.QC 20200120