General Bounds on the Mutual Information Between a Parameter and n Conditionally Independent Observations

Each parameter ` in an abstract parameter space \Theta is associated with a different probability distribution on a set Y . A parameter ` is chosen at random from \Theta according to some a priori distribution on \Theta, and n conditionally independent random variables Y n = Y 1 ; : : : Yn are observed with common distribution determined by `. We obtain bounds on the mutual information between the random variable \Theta, giving the choice of parameter, and the random variable Y n , giving the sequence of observations. We also bound the supremum of the mutual information, over choices of the prior distribution on \Theta. These quantities have applications in density estimation, computational learning theory, universal coding, hypothesis testing, and portfolio selection theory. The bounds are given in terms of the metric and information dimensions of the parameter space \Theta with respect to the Hellinger distance. 1 Introduction Let us assume that the state of nature is defined ..