Surprising properties of dropout in deep networks
  We analyze dropout in deep networks with rectified linear units and the
quadratic loss. Our results expose surprising differences between the behavior
of dropout and more traditional regularizers like weight decay. For example, on
some simple data sets dropout training produces negative weights even though
the output is the sum of the inputs. This provides a counterpoint to the
suggestion that dropout discourages co-adaptation of weights. We also show that
the dropout penalty can grow exponentially in the depth of the network while
the weight-decay penalty remains essentially linear, and that dropout is
insensitive to various re-scalings of the input features, outputs, and network
weights. This last insensitivity implies that there are no isolated local
minima of the dropout training criterion. Our work uncovers new properties of
dropout, extends our understanding of why dropout succeeds, and lays the
foundation for further progress.
