Efficient Decentralized Deep Learning by Dynamic Model Averaging
  We propose an efficient protocol for decentralized training of deep neural
networks from distributed data sources. The proposed protocol allows to handle
different phases of model training equally well and to quickly adapt to concept
drifts. This leads to a reduction of communication by an order of magnitude
compared to periodically communicating state-of-the-art approaches. Moreover,
we derive a communication bound that scales well with the hardness of the
serialized learning problem. The reduction in communication comes at almost no
cost, as the predictive performance remains virtually unchanged. Indeed, the
proposed protocol retains loss bounds of periodically averaging schemes. An
extensive empirical evaluation validates major improvement of the trade-off
between model performance and communication which could be beneficial for
numerous decentralized learning applications, such as autonomous driving, or
voice recognition and image classification on mobile phones.
