Error Feedback Fixes SignSGD and other Gradient Compression Schemes
  Sign-based algorithms (e.g. signSGD) have been proposed as a biased gradient
compression technique to alleviate the communication bottleneck in training
large neural networks across multiple workers. We show simple convex
counter-examples where signSGD does not converge to the optimum. Further, even
when it does converge, signSGD may generalize poorly when compared with SGD.
These issues arise because of the biased nature of the sign compression
operator. We then show that using error-feedback, i.e. incorporating the error
made by the compression operator into the next step, overcomes these issues. We
prove that our algorithm EF-SGD with arbitrary compression operator achieves
the same rate of convergence as SGD without any additional assumptions. Thus
EF-SGD achieves gradient compression for free. Our experiments thoroughly
substantiate the theory and show that error-feedback improves both convergence
and generalization. Code can be found at
\url{https://github.com/epfml/error-feedback-SGD}.
