Probabilistic Binary Neural Networks
  Low bit-width weights and activations are an effective way of combating the
increasing need for both memory and compute power of Deep Neural Networks. In
this work, we present a probabilistic training method for Neural Network with
both binary weights and activations, called BLRNet. By embracing stochasticity
during training, we circumvent the need to approximate the gradient of
non-differentiable functions such as sign(), while still obtaining a fully
Binary Neural Network at test time. Moreover, it allows for anytime ensemble
predictions for improved performance and uncertainty estimates by sampling from
the weight distribution. Since all operations in a layer of the BLRNet operate
on random variables, we introduce stochastic versions of Batch Normalization
and max pooling, which transfer well to a deterministic network at test time.
We evaluate the BLRNet on multiple standardized benchmarks.
