Bit-wise Training of Neural Network Weights
  We introduce an algorithm where the individual bits representing the weights
of a neural network are learned. This method allows training weights with
integer values on arbitrary bit-depths and naturally uncovers sparse networks,
without additional constraints or regularization techniques. We show better
results than the standard training technique with fully connected networks and
similar performance as compared to standard training for convolutional and
residual networks. By training bits in a selective manner we found that the
biggest contribution to achieving high accuracy is given by the first three
most significant bits, while the rest provide an intrinsic regularization. As a
consequence more than 90\% of a network can be used to store arbitrary codes
without affecting its accuracy. These codes may be random noise, binary files
or even the weights of previously trained networks.
