Matrix Backpropagation for Deep Networks with Structured Layers

Deep neural network architectures have recently pro-duced excellent results in a variety of areas in artificial intelligence and visual recognition, well surpassing tradi-tional shallow architectures trained using hand-designed features. The power of deep networks stems both from their ability to perform local computations followed by pointwise non-linearities over increasingly larger receptive fields, and from the simplicity and scalability of the gradient-descent training procedure based on backpropagation. An open problem is the inclusion of layers that perform global, struc-tured matrix computations like segmentation (e.g. normal-ized cuts) or higher-order pooling (e.g. log-tangent space metrics defined over the manifold of symmetric positive def-inite matrices) while preserving the validity and efficiency of an end-to-end deep training framework. In this paper we propose a sound mathematical apparatus to formally inte-grate global structured computation into deep computation architectures. At the heart of our methodology is the de-velopment of the theory and practice of backpropagation that generalizes to the calculus of adjoint matrix variations. We perform segmentation experiments using the BSDS and MSCOCO benchmarks and demonstrate that deep networks relying on second-order pooling and normalized cuts lay-ers, trained end-to-end using matrix backpropagation, out-perform counterparts that do not take advantage of such global layers. 1