Zero-bias autoencoders and the benefits of co-adapting features

We show that training common regularized autoencoders resembles clustering, because it amounts to fitting a density model whose mass is concentrated in the directions of the individ-ual weight vectors. We then propose a new ac-tivation function based on thresholding a linear function with zero bias (so it is truly linear not affine), and argue that this allows hidden units to “collaborate ” in order to define larger regions of uniform density. We show that the new activa-tion function makes it possible to train autoen-coders without an explicit regularization penalty, such as sparsification, contraction or denoising, by simply minimizing reconstruction error. Ex-periments in a variety of recognition tasks show that zero-bias autoencoders perform about on par with common regularized autoencoders on low dimensional data and outperform these by an in-creasing margin as the dimensionality of the data increases. 1