Lecture 2

In the last lecture, we introduced entropy H(X), and conditional entry H(X|Y), and showed how they are related via the chain rule. We also proved the inequality H(X1,..., Xn) ≤ H(X1) + · · · + H(Xn). In this lecture, we will derive two more useful inequalities and then give some examples where they are useful. 1 More Inequalities Our first inequality shows that conditioning can only reduce the entropy. This is in contrast to general probabilities (where conditioning can either decrease or increase) as well as some of the quantities we look look at in the next lecture such as the mutual information