Noise Injection: theoretical prospects

Noise Injection consists in adding noise to the inputs during neural network training. Experimental results suggest that it might improve the generalization ability of the resulting neural network. A justification of this improvement remains elusive: first, describing analytically the average perturbed cost function is difficult, second, controlling the fluctuations of the random perturbed cost function is hard. Hence recent papers suggest to replace the random perturbed cost by a (deterministic) Taylor approximation of the average perturbed cost function. This paper takes a different stance: when the injected noise is Gaussian, Noise Injection is naturally connected to the action of the Heat Kernel. This provides indications on the relevance domain of traditional Taylor expansions, and shows the dependence of the quality of Taylor approximations on global smoothness properties of neural networks under consideration. The connection between noise injection and Heat kernel also enables t..