Pushing the bounds of dropout
  We show that dropout training is best understood as performing MAP estimation
concurrently for a family of conditional models whose objectives are themselves
lower bounded by the original dropout objective. This discovery allows us to
pick any model from this family after training, which leads to a substantial
improvement on regularisation-heavy language modelling. The family includes
models that compute a power mean over the sampled dropout masks, and their less
stochastic subvariants with tighter and higher lower bounds than the fully
stochastic dropout objective. We argue that since the deterministic
subvariant's bound is equal to its objective, and the highest amongst these
models, the predominant view of it as a good approximation to MC averaging is
misleading. Rather, deterministic dropout is the best available approximation
to the true objective.
