On dynamic programming-like recursive gradient formula for alleviating MLP hidden-node saturation in the parity problem

[[abstract]]This paper addresses saturation phenomena at hidden nodes during the learning phase of neural networks. The hidden-node saturation tends to cause a "plateau," a region of very little or no change in a graphic representation of the error learning curve. We investigate the saturation phenomena in multilayer perceptrons (MLP) with the well-known neuralnetwork benchmark problem: the parity problem, describing how to augment their learning capacity to solve it perfectly by avoiding the hidden-node saturation in conjunction with a dynamic programminglike recursive gradient formula. To make the parity problem especially challenging, we first show that the seven-bit parity problem can in principle be solved using only four hidden nodes and we then compare various learning algorithms using this MLP architecture with emphasis on whether or not each algorithm can solve the problem "perfectly" regardless of its learning speed. In particular, we highlight an onlinemode steepest descent-type learnin[[fileno]]2030226030076[[department]]資訊工程學