Comparison Of Maximum Entropy And Minimal Mutual Information In A Nonlinear Setting

In blind sourc separation (BSS), two di#erent separation tecration are mainly used: minimal mutual information (MMI), where minimization of the mutual output information yields an independent random vecomu and maximum entropy (ME), where the output entropy is maximized. However, it is yetuncz'z why ME should solve the separation problem, i.e. result in an independentvecepe Yang and Amari have given a partialctialuHz:A for ME in the linearcne in [18], where they prove that under the assumption of vanishingexpecnguz# of thesourcY ME does notctuYM the solutions of MMIexc[A forscuMM# and permutation. In this paper, we generalize Yang and Amari's approac to nonlinear BSS problems, where random vecomu are mixed by output func:z:u of layered neural networks. We show thatcatuYH solution points of MMI are kept #xed by ME if noscz:[z in all layers is allowed. In general, ME, however, might alsocsouY the scuA#: in the non-output network layers,hencs leaving the MMI solution points. Therefore, wecuYzHzH this paper by suggesting that in nonlinear ME algorithms, the norm of all weight matrix rows ofeac non-output layer should be kept #xed in laterepocM during network training. ? 2002 ElsevierScevie B.V. All rights reserved