Quantal synaptic dilution enhances sparse encoding and dropout
  regularisation in deep networks
  Dropout is a technique that silences the activity of units stochastically
while training deep networks to reduce overfitting. Here we introduce Quantal
Synaptic Dilution (QSD), a biologically plausible model of dropout
regularisation based on the quantal properties of neuronal synapses, that
incorporates heterogeneities in response magnitudes and release probabilities
for vesicular quanta. QSD outperforms standard dropout in ReLU multilayer
perceptrons, with enhanced sparse encoding at test time when dropout masks are
replaced with identity functions, without shifts in trainable weight or bias
distributions. For convolutional networks, the method also improves
generalisation in computer vision tasks with and without inclusion of
additional forms of regularisation. QSD also outperforms standard dropout in
recurrent networks for language modelling and sentiment analysis. An advantage
of QSD over many variations of dropout is that it can be implemented generally
in all conventional deep networks where standard dropout is applicable.
