Distributed deep learning on edge-devices: feasibility via adaptive
  compression
  A large portion of data mining and analytic services use modern machine
learning techniques, such as deep learning. The state-of-the-art results by
deep learning come at the price of an intensive use of computing resources. The
leading frameworks (e.g., TensorFlow) are executed on GPUs or on high-end
servers in datacenters. On the other end, there is a proliferation of personal
devices with possibly free CPU cycles; this can enable services to run in
users' homes, embedding machine learning operations. In this paper, we ask the
following question: Is distributed deep learning computation on WAN connected
devices feasible, in spite of the traffic caused by learning tasks? We show
that such a setup rises some important challenges, most notably the ingress
traffic that the servers hosting the up-to-date model have to sustain.
  In order to reduce this stress, we propose adaComp, a novel algorithm for
compressing worker updates to the model on the server. Applicable to stochastic
gradient descent based approaches, it combines efficient gradient selection and
learning rate modulation. We then experiment and measure the impact of
compression, device heterogeneity and reliability on the accuracy of learned
models, with an emulator platform that embeds TensorFlow into Linux containers.
We report a reduction of the total amount of data sent by workers to the server
by two order of magnitude (e.g., 191-fold reduction for a convolutional network
on the MNIST dataset), when compared to a standard asynchronous stochastic
gradient descent, while preserving model accuracy.
