FTPipeHD: A Fault-Tolerant Pipeline-Parallel Distributed Training
  Framework for Heterogeneous Edge Devices
  With the increased penetration and proliferation of Internet of Things (IoT)
devices, there is a growing trend towards distributing the power of deep
learning (DL) across edge devices rather than centralizing it in the cloud.
This development enables better privacy preservation, real-time responses, and
user-specific models. To deploy deep and complex models to edge devices with
limited resources, model partitioning of deep neural networks (DNN) model is
necessary, and has been widely studied. However, most of the existing
literature only considers distributing the inference model while still relying
centralized cloud infrastructure to generate this model through training. In
this paper, we propose FTPipeHD, a novel DNN training framework that trains DNN
models across distributed heterogeneous devices with fault tolerance mechanism.
To accelerate the training with time-varying computing power of each device, we
optimize the partition points dynamically according to real-time computing
capacities. We also propose a novel weight redistribution approach that
replicates the weights to both the neighboring nodes and the central node
periodically, which combats the failure of multiple devices during training
while incurring limited communication cost. Our numerical results demonstrate
that FTPipeHD is 6.8x faster in training than the state of the art method when
the computing capacity of the best device is 10x greater than the worst one. It
is also shown that the proposed method is able to accelerate the training even
with the existence of device failures.
