Bitwise Neural Networks

Based on the assumption that there exists a neu-ral network that efficiently represents a set of Boolean functions between all binary inputs and outputs, we propose a process for developing and deploying neural networks whose weight param-eters, bias terms, input, and intermediate hid-den layer output signals, are all binary-valued, and require only basic bit logic for the feedfor-ward pass. The proposed Bitwise Neural Net-work (BNN) is especially suitable for resource-constrained environments, since it replaces ei-ther floating or fixed-point arithmetic with signif-icantly more efficient bitwise operations. Hence, the BNN requires for less spatial complexity, less memory bandwidth, and less power consumption in hardware. In order to design such networks, we propose to add a few training schemes, such as weight compression and noisy backpropaga-tion, which result in a bitwise network that per-forms almost as well as its corresponding real-valued network. We test the proposed network on the MNIST dataset, represented using binary features, and show that BNNs result in compet-itive performance while offering dramatic com-putational savings. 1