Enabling On-Device CNN Training by Self-Supervised Instance Filtering
  and Error Map Pruning
  This work aims to enable on-device training of convolutional neural networks
(CNNs) by reducing the computation cost at training time. CNN models are
usually trained on high-performance computers and only the trained models are
deployed to edge devices. But the statically trained model cannot adapt
dynamically in a real environment and may result in low accuracy for new
inputs. On-device training by learning from the real-world data after
deployment can greatly improve accuracy. However, the high computation cost
makes training prohibitive for resource-constrained devices. To tackle this
problem, we explore the computational redundancies in training and reduce the
computation cost by two complementary approaches: self-supervised early
instance filtering on data level and error map pruning on the algorithm level.
The early instance filter selects important instances from the input stream to
train the network and drops trivial ones. The error map pruning further prunes
out insignificant computations when training with the selected instances.
Extensive experiments show that the computation cost is substantially reduced
without any or with marginal accuracy loss. For example, when training
ResNet-110 on CIFAR-10, we achieve 68% computation saving while preserving full
accuracy and 75% computation saving with a marginal accuracy loss of 1.3%.
Aggressive computation saving of 96% is achieved with less than 0.1% accuracy
loss when quantization is integrated into the proposed approaches. Besides,
when training LeNet on MNIST, we save 79% computation while boosting accuracy
by 0.2%.
