ASNI: Adaptive Structured Noise Injection for shallow and deep neural
  networks
  Dropout is a regularisation technique in neural network training where unit
activations are randomly set to zero with a given probability
\emph{independently}. In this work, we propose a generalisation of dropout and
other multiplicative noise injection schemes for shallow and deep neural
networks, where the random noise applied to different units is not independent
but follows a joint distribution that is either fixed or estimated during
training. We provide theoretical insights on why such adaptive structured noise
injection (ASNI) may be relevant, and empirically confirm that it helps boost
the accuracy of simple feedforward and convolutional neural networks,
disentangles the hidden layer representations, and leads to sparser
representations. Our proposed method is a straightforward modification of the
classical dropout and does not require additional computational overhead.
