Methods and Applications of Deep Neural Networks

Neural networks are universal function approximators and have been widely used in performing tasks for artificial intelligence. Despite their generality, neural networks are also known to be hard to harness due to their complicated mathematical nature and the sophistication of an application domain. In this thesis, we first address neural network training. Classical optimization literature often fails to provide effective algorithms in practice. This is because the optimization problems associated to neural networks are difficult for their non-linearity and non-convexity. We propose to solve two problems in neural network training: vanishing/exploding gradients and scalability of second-order methods. For each of the problem, we provide a principled approach and provable results. Then, we look at an application of neural networks in computer vision, optical flow estimation. This application was often address with classical optimization techniques such as Markov random fields. However, neural networks when fueled with sufficient training data often outperform the classical techniques. We propose a novel neural network model for optical flow estimation with the principle to solve the "small things moving fast" problem. Experiments on both synthetic and real-world datasets are performed to demonstrate the above methods