Function approximation in Hilbert spaces: a general sequential method and a particular implementation with neural networks

A sequential method for approximating vectors in Hilbert spaces, called Sequential Approximation with Optimal Coefficients (SAOC), is presented. Most of the existing sequential methods choose the new term so that it matches the previous residue as best as possible. Although this strategy leads to approximations convergent towards the target function, it may be far from being the best strategy with regard to the number of terms of the approximation. SAOC combines two key ideas. The first is the optimization of the coefficients (the linear part of the approximation). The second is the flexibility to choose the frequencies (the nonlinear part). The only relation with the residue has to do with its approximation capability of the target vector $f$. SAOC maintains orthogonal-like properties. The theoretical results obtained proof that, under reasonable conditions, the construction of the approximation is always possible and, in the limit, the residue of the approximation obtained with SAOC is the best one that can be obtained with any subset of the given set of vectors. In addition, it seems that it should achieve the same accuracy that other existent sequential methods with fewer terms. In the particular case of $L^2$, it can be applied to polynomials, Fourier series, wavelets and neural networks, among others. Also, a particular implementation using neural networks is presented. In fact, the profit is reciprocal, because SAOC can be used as an inspiration to construct and train a neural network