Dropout with Tabu Strategy for Regularizing Deep Neural Networks
  Dropout has proven to be an effective technique for regularization and
preventing the co-adaptation of neurons in deep neural networks (DNN). It
randomly drops units with a probability $p$ during the training stage of DNN.
Dropout also provides a way of approximately combining exponentially many
different neural network architectures efficiently. In this work, we add a
diversification strategy into dropout, which aims at generating more different
neural network architectures in a proper times of iterations. The dropped units
in last forward propagation will be marked. Then the selected units for
dropping in the current FP will be kept if they have been marked in the last
forward propagation. We only mark the units from the last forward propagation.
We call this new technique Tabu Dropout. Tabu Dropout has no extra parameters
compared with the standard Dropout and also it is computationally cheap. The
experiments conducted on MNIST, Fashion-MNIST datasets show that Tabu Dropout
improves the performance of the standard dropout.
