Function aproximation with SAOCIF: a general sequential method and a particular algorithm with feed-forward neural networks

A sequential method for approximating vectors in Hilbert spaces, called Sequential Approximation with Optimal Coefficients and Interacting Frequencies (SAOCIF), is presented. SAOCIF combines two key ideas. The first one is the optimization of the coefficients (the linear part of the approximation). The second one is the flexibility to choose the frequencies (the non-linear part). The only relation with the previous residue has to do with its approximation capability of the target vector f. The approximations defined by SAOCIF always exist, and maintain orthogonal-like properties. The theoretical results obtained prove that, under reasonable conditions, the residue of the approximation obtained with SAOCIF (in the limit) is the best one that can be obtained with any subset of the given set of vectors. In the particular case of L^2, it can be applied to approximations by algebraic polynomials, Fourier series, wavelets and feed-forward neural networks, among others. Also, a particular algorithm with neural networks is presented. The resulting method combines the locality of sequential approximations, where only one frequency is found at every step, with the globality of non-sequential methods, such as Backpropagation, where every frequency interacts with the others. Experimental results show a very satisfactory performance of this new method and several suggesting ideas for future experiments