Neuron-Specific Dropout: A Deterministic Regularization Technique to
  Prevent Neural Networks from Overfitting & Reduce Dependence on Large
  Training Samples
  In order to develop complex relationships between their inputs and outputs,
deep neural networks train and adjust large number of parameters. To make these
networks work at high accuracy, vast amounts of data are needed. Sometimes,
however, the quantity of data needed is not present or obtainable for training.
Neuron-specific dropout (NSDropout) is a tool to address this problem.
NSDropout looks at both the training pass, and validation pass, of a layer in a
model. By comparing the average values produced by each neuron for each class
in a data set, the network is able to drop targeted units. The layer is able to
predict what features, or noise, the model is looking at during testing that
isn't present when looking at samples from validation. Unlike dropout, the
"thinned" networks cannot be "unthinned" for testing. Neuron-specific dropout
has proved to achieve similar, if not better, testing accuracy with far less
data than traditional methods including dropout and other regularization
methods. Experimentation has shown that neuron-specific dropout reduces the
chance of a network overfitting and reduces the need for large training samples
on supervised learning tasks in image recognition, all while producing
best-in-class results.
