Memorized Sparse Backpropagation
  Neural network learning is usually time-consuming since backpropagation needs
to compute full gradients and backpropagate them across multiple layers.
Despite its success of existing works in accelerating propagation through
sparseness, the relevant theoretical characteristics remain under-researched
and empirical studies found that they suffer from the loss of information
contained in unpropagated gradients. To tackle these problems, this paper
presents a unified sparse backpropagation framework and provides a detailed
analysis of its theoretical characteristics. Analysis reveals that when applied
to a multilayer perceptron, our framework essentially performs gradient descent
using an estimated gradient similar enough to the true gradient, resulting in
convergence in probability under certain conditions. Furthermore, a simple yet
effective algorithm named memorized sparse backpropagation (MSBP) is proposed
to remedy the problem of information loss by storing unpropagated gradients in
memory for learning in the next steps. Experimental results demonstrate that
the proposed MSBP is effective to alleviate the information loss in traditional
sparse backpropagation while achieving comparable acceleration.
