On the influence of roundoff errors on the convergence of the gradient
  descent method with low-precision floating-point computation
  The employment of stochastic rounding schemes helps prevent stagnation of
convergence, due to vanishing gradient effect when implementing the gradient
descent method in low precision. Conventional stochastic rounding achieves zero
bias by preserving small updates with probabilities proportional to their
relative magnitudes. In this study, we propose a new stochastic rounding scheme
that trades the zero bias property with a larger probability to preserve small
gradients. Our method yields a constant rounding bias that, at each iteration,
lies in a descent direction. For convex problems, we prove that the proposed
rounding method has a beneficial effect on the convergence rate of gradient
descent. We validate our theoretical analysis by comparing the performances of
various rounding schemes when optimizing a multinomial logistic regression
model and when training a simple neural network with 8-bit floating-point
format.
