Energy-based Dropout in Restricted Boltzmann Machines: Why not go random
  Deep learning architectures have been widely fostered throughout the last
years, being used in a wide range of applications, such as object recognition,
image reconstruction, and signal processing. Nevertheless, such models suffer
from a common problem known as overfitting, which limits the network from
predicting unseen data effectively. Regularization approaches arise in an
attempt to address such a shortcoming. Among them, one can refer to the
well-known Dropout, which tackles the problem by randomly shutting down a set
of neurons and their connections according to a certain probability. Therefore,
this approach does not consider any additional knowledge to decide which units
should be disconnected. In this paper, we propose an energy-based Dropout
(E-Dropout) that makes conscious decisions whether a neuron should be dropped
or not. Specifically, we design this regularization method by correlating
neurons and the model's energy as an importance level for further applying it
to energy-based models, such as Restricted Boltzmann Machines (RBMs). The
experimental results over several benchmark datasets revealed the proposed
approach's suitability compared to the traditional Dropout and the standard
RBMs.
