The Flip Side of the Reweighted Coin: Duality of Adaptive Dropout and
  Regularization
  Among the most successful methods for sparsifying deep (neural) networks are
those that adaptively mask the network weights throughout training. By
examining this masking, or dropout, in the linear case, we uncover a duality
between such adaptive methods and regularization through the so-called
"$\eta$-trick" that casts both as iteratively reweighted optimizations. We show
that any dropout strategy that adapts to the weights in a monotonic way
corresponds to an effective subquadratic regularization penalty, and therefore
leads to sparse solutions. We obtain the effective penalties for several
popular sparsification strategies, which are remarkably similar to classical
penalties commonly used in sparse optimization. Considering variational dropout
as a case study, we demonstrate similar empirical behavior between the adaptive
dropout method and classical methods on the task of deep network
sparsification, validating our theory.
