Sign Bit is Enough: A Learning Synchronization Framework for Multi-hop
  All-reduce with Ultimate Compression
  Traditional one-bit compressed stochastic gradient descent can not be
directly employed in multi-hop all-reduce, a widely adopted distributed
training paradigm in network-intensive high-performance computing systems such
as public clouds. According to our theoretical findings, due to the cascading
compression, the training process has considerable deterioration on the
convergence performance. To overcome this limitation, we implement a sign-bit
compression-based learning synchronization framework, Marsit. It prevents
cascading compression via an elaborate bit-wise operation for unbiased sign
aggregation and its specific global compensation mechanism for mitigating
compression deviation. The proposed framework retains the same theoretical
convergence rate as non-compression mechanisms. Experimental results
demonstrate that Marsit reduces up to 35% training time while preserving the
same accuracy as training without compression.
