Finding Compact and Sparse Distributed Representations of Visual Images

Some recent work has investigated the dichotomy between compact coding using dimensionality reduction and sparse distributed coding in the context of understanding biological information processing. We introduce an artificial neural network which self organises on the basis of simple Hebbian learning and negative feedback of activation and show that it is capable of both forming compact codings of data distributions and also of identifying filters most sensitive to sparse distributed codes. The network is extremely simple and its biological relevance is investigated via its response to a set of images which are typical of everyday life. However, an analysis of the network's identification of the filter for sparse coding reveals that this coding may not be globally optimal and that there exists an innate limiting factor which cannot be transcended. 1 Introduction Recently, Field [2] has made an important distinction between a "compact code" and a "sparse distributed code". A compact c..