MaxDropout: Deep Neural Network Regularization Based on Maximum Output
  Values
  Different techniques have emerged in the deep learning scenario, such as
Convolutional Neural Networks, Deep Belief Networks, and Long Short-Term Memory
Networks, to cite a few. In lockstep, regularization methods, which aim to
prevent overfitting by penalizing the weight connections, or turning off some
units, have been widely studied either. In this paper, we present a novel
approach called MaxDropout, a regularizer for deep neural network models that
works in a supervised fashion by removing (shutting off) the prominent neurons
(i.e., most active) in each hidden layer. The model forces fewer activated
units to learn more representative information, thus providing sparsity.
Regarding the experiments, we show that it is possible to improve existing
neural networks and provide better results in neural networks when Dropout is
replaced by MaxDropout. The proposed method was evaluated in image
classification, achieving comparable results to existing regularizers, such as
Cutout and RandomErasing, also improving the accuracy of neural networks that
uses Dropout by replacing the existing layer by MaxDropout.
