Sparse Weight Activation Training
  Neural network training is computationally and memory intensive. Sparse
training can reduce the burden on emerging hardware platforms designed to
accelerate sparse computations, but it can affect network convergence. In this
work, we propose a novel CNN training algorithm Sparse Weight Activation
Training (SWAT). SWAT is more computation and memory-efficient than
conventional training. SWAT modifies back-propagation based on the empirical
insight that convergence during training tends to be robust to the elimination
of (i) small magnitude weights during the forward pass and (ii) both small
magnitude weights and activations during the backward pass. We evaluate SWAT on
recent CNN architectures such as ResNet, VGG, DenseNet and WideResNet using
CIFAR-10, CIFAR-100 and ImageNet datasets. For ResNet-50 on ImageNet SWAT
reduces total floating-point operations (FLOPS) during training by 80%
resulting in a 3.3$\times$ training speedup when run on a simulated sparse
learning accelerator representative of emerging platforms while incurring only
1.63% reduction in validation accuracy. Moreover, SWAT reduces memory footprint
during the backward pass by 23% to 50% for activations and 50% to 90% for
weights.
