Avoiding pathologies in very deep networks

Choosing appropriate architectures and regular-ization strategies for deep networks is crucial to good predictive performance. To shed light on this problem, we analyze the analogous prob-lem of constructing useful priors on composi-tions of functions. Specifically, we study the deep Gaussian process, a type of infinitely-wide, deep neural network. We show that in standard architectures, the representational capacity of the network tends to capture fewer degrees of free-dom as the number of layers increases, retaining only a single degree of freedom in the limit. We propose an alternate network architecture which does not suffer from this pathology. We also examine deep covariance functions, obtained by composing infinitely many feature transforms. Lastly, we characterize the class of models ob-tained by performing dropout on Gaussian pro-cesses.