SGAD: Soft-Guided Adaptively-Dropped Neural Network
  Deep neural networks (DNNs) have been proven to have many redundancies.
Hence, many efforts have been made to compress DNNs. However, the existing
model compression methods treat all the input samples equally while ignoring
the fact that the difficulties of various input samples being correctly
classified are different. To address this problem, DNNs with adaptive dropping
mechanism are well explored in this work. To inform the DNNs how difficult the
input samples can be classified, a guideline that contains the information of
input samples is introduced to improve the performance. Based on the developed
guideline and adaptive dropping mechanism, an innovative soft-guided
adaptively-dropped (SGAD) neural network is proposed in this paper. Compared
with the 32 layers residual neural networks, the presented SGAD can reduce the
FLOPs by 77% with less than 1% drop in accuracy on CIFAR-10.
