On the Inductive Bias of Dropout
  Dropout is a simple but effective technique for learning in neural networks
and other settings. A sound theoretical understanding of dropout is needed to
determine when dropout should be applied and how to use it most effectively. In
this paper we continue the exploration of dropout as a regularizer pioneered by
Wager, et.al. We focus on linear classification where a convex proxy to the
misclassification loss (i.e. the logistic loss used in logistic regression) is
minimized. We show: (a) when the dropout-regularized criterion has a unique
minimizer, (b) when the dropout-regularization penalty goes to infinity with
the weights, and when it remains bounded, (c) that the dropout regularization
can be non-monotonic as individual weights increase from 0, and (d) that the
dropout regularization penalty may not be convex. This last point is
particularly surprising because the combination of dropout regularization with
any convex loss proxy is always a convex function.
  In order to contrast dropout regularization with $L_2$ regularization, we
formalize the notion of when different sources are more compatible with
different regularizers. We then exhibit distributions that are provably more
compatible with dropout regularization than $L_2$ regularization, and vice
versa. These sources provide additional insight into how the inductive biases
of dropout and $L_2$ regularization differ. We provide some similar results for
$L_1$ regularization.
