Continuous Dropout
  Dropout has been proven to be an effective algorithm for training robust deep
networks because of its ability to prevent overfitting by avoiding the
co-adaptation of feature detectors. Current explanations of dropout include
bagging, naive Bayes, regularization, and sex in evolution. According to the
activation patterns of neurons in the human brain, when faced with different
situations, the firing rates of neurons are random and continuous, not binary
as current dropout does. Inspired by this phenomenon, we extend the traditional
binary dropout to continuous dropout. On the one hand, continuous dropout is
considerably closer to the activation characteristics of neurons in the human
brain than traditional binary dropout. On the other hand, we demonstrate that
continuous dropout has the property of avoiding the co-adaptation of feature
detectors, which suggests that we can extract more independent feature
detectors for model averaging in the test stage. We introduce the proposed
continuous dropout to a feedforward neural network and comprehensively compare
it with binary dropout, adaptive dropout, and DropConnect on MNIST, CIFAR-10,
SVHN, NORB, and ILSVRC-12. Thorough experiments demonstrate that our method
performs better in preventing the co-adaptation of feature detectors and
improves test performance. The code is available at:
https://github.com/jasonustc/caffe-multigpu/tree/dropout.
