LayerPipe: Accelerating Deep Neural Network Training by Intra-Layer and
  Inter-Layer Gradient Pipelining and Multiprocessor Scheduling
  The time required for training the neural networks increases with size,
complexity, and depth. Training model parameters by backpropagation inherently
creates feedback loops. These loops hinder efficient pipelining and scheduling
of the tasks within the layer and between consecutive layers. Prior approaches,
such as PipeDream, have exploited the use of delayed gradient to achieve
inter-layer pipelining. However, these approaches treat the entire
backpropagation as a single task; this leads to an increase in computation time
and processor underutilization. This paper presents novel optimization
approaches where the gradient computations with respect to the weights and the
activation functions are considered independently; therefore, these can be
computed in parallel. This is referred to as intra-layer optimization.
Additionally, the gradient computation with respect to the activation function
is further divided into two parts and distributed to two consecutive layers.
This leads to balanced scheduling where the computation time of each layer is
the same. This is referred to as inter-layer optimization. The proposed system,
referred to as LayerPipe, reduces the number of clock cycles required for
training while maximizing processor utilization with minimal inter-processor
communication overhead. LayerPipe achieves an average speedup of 25% and
upwards of 80% with 7 to 9 processors with less communication overhead when
compared to PipeDream.
