Deep Networks with Fast Retraining
  Recent work [1] has utilized Moore-Penrose (MP) inverse in deep convolutional
neural network (DCNN) learning, which achieves better generalization
performance over the DCNN with a stochastic gradient descent (SGD) pipeline.
However, Yang's work has not gained much popularity in practice due to its high
sensitivity of hyper-parameters and stringent demands of computational
resources. To enhance its applicability, this paper proposes a novel MP
inverse-based fast retraining strategy. In each training epoch, a random
learning strategy that controls the number of convolutional layers trained in
the backward pass is first utilized. Then, an MP inverse-based batch-by-batch
learning strategy, which enables the network to be implemented without access
to industrial-scale computational resources, is developed to refine the dense
layer parameters. Experimental results empirically demonstrate that fast
retraining is a unified strategy that can be used for all DCNNs. Compared to
other learning strategies, the proposed learning pipeline has robustness
against the hyper-parameters, and the requirement of computational resources is
significantly reduced. [1] Y. Yang, J. Wu, X. Feng, and A. Thangarajah,
"Recomputation of dense layers for the perfor-238mance improvement of dcnn,"
IEEE Trans. Pattern Anal. Mach. Intell., 2019.
