Weight Update Skipping: Reducing Training Time for Artificial Neural
  Networks
  Artificial Neural Networks (ANNs) are known as state-of-the-art techniques in
Machine Learning (ML) and have achieved outstanding results in data-intensive
applications, such as recognition, classification, and segmentation. These
networks mostly use deep layers of convolution or fully connected layers with
many filters in each layer, demanding a large amount of data and tunable
hyperparameters to achieve competitive accuracy. As a result, storage,
communication, and computational costs of training (in particular training
time) become limiting factors to scale them up. In this paper, we propose a new
training methodology for ANNs that exploits the observation of improvement of
accuracy shows temporal variations which allow us to skip updating weights when
the variation is minuscule. During such time windows, we keep updating bias
which ensures the network still trains and avoids overfitting; however, we
selectively skip updating weights (and their time-consuming computations). Such
a training approach virtually achieves the same accuracy with considerably less
computational cost, thus lower training time. We propose two methods for
updating weights and evaluate them by analyzing four state-of-the-art models,
AlexNet, VGG-11, VGG-16, ResNet-18 on CIFAR datasets. On average, our two
proposed methods called WUS and WUS+LR reduced the training time (compared to
the baseline) by 54%, and 50%, respectively on CIFAR-10; and 43% and 35% on
CIFAR-100, respectively.
