On the Mathematical Foundation of Learning Algorithms

A general mathematical framework is developed for learning algorithms. A learning task belongs to either of two classes, reactive and adaptive. For the reactive tasks, learning algorithms can be analysed as statistical estimators, for which the theory of Bayesian information geometry provides a complete description. In particular, there exists ideal estimates holding all the information in the sample. Under computational constraint the optimal estimates are obtained by approximating the ideal estimates. This encompasses most of the commonly used statistical principles and criteria. For the adaptive tasks no complete theory exists at present, but the results for reactive tasks can be used as both components and guidance. Several most important special cases are analysed. Key words: Bayesian information geometry, ideal estimates, Gaussian processes, learning theory Running head: Foundation of Learning Contents 1 Introduction 1 2 Background on Information Geometry 3 2.1 The basic prob..