Reintroducing Straight-Through Estimators as Principled Methods for
  Stochastic Binary Networks
  Training neural networks with binary weights and activations is a challenging
problem due to the lack of gradients and difficulty of optimization over
discrete weights. Many successful experimental results have been achieved with
empirical straight-through (ST) approaches, proposing a variety of ad-hoc rules
for propagating gradients through non-differentiable activations and updating
discrete weights. At the same time, ST methods can be truly derived as
estimators in the stochastic binary network (SBN) model with Bernoulli weights.
We advance these derivations to a more complete and systematic study. We
analyze properties, estimation accuracy, obtain different forms of correct ST
estimators for activations and weights, explain existing empirical approaches
and their shortcomings, explain how latent weights arise from the mirror
descent method when optimizing over probabilities. This allows to reintroduce
ST methods, long known empirically, as sound approximations, apply them with
clarity and develop further improvements.
