Dropout as a Structured Shrinkage Prior
  Dropout regularization of deep neural networks has been a mysterious yet
effective tool to prevent overfitting. Explanations for its success range from
the prevention of "co-adapted" weights to it being a form of cheap Bayesian
inference. We propose a novel framework for understanding multiplicative noise
in neural networks, considering continuous distributions as well as Bernoulli
noise (i.e. dropout). We show that multiplicative noise induces structured
shrinkage priors on a network's weights. We derive the equivalence through
reparametrization properties of scale mixtures and without invoking any
approximations. Given the equivalence, we then show that dropout's Monte Carlo
training objective approximates marginal MAP estimation. We leverage these
insights to propose a novel shrinkage framework for resnets, terming the prior
'automatic depth determination' as it is the natural analog of automatic
relevance determination for network depth. Lastly, we investigate two inference
strategies that improve upon the aforementioned MAP approximation in regression
benchmarks.
