Probabilistic fine-tuning of pruning masks and PAC-Bayes self-bounded
  learning
  We study an approach to learning pruning masks by optimizing the expected
loss of stochastic pruning masks, i.e., masks which zero out each weight
independently with some weight-specific probability. We analyze the training
dynamics of the induced stochastic predictor in the setting of linear
regression, and observe a data-adaptive L1 regularization term, in contrast to
the dataadaptive L2 regularization term known to underlie dropout in linear
regression. We also observe a preference to prune weights that are less
well-aligned with the data labels. We evaluate probabilistic fine-tuning for
optimizing stochastic pruning masks for neural networks, starting from masks
produced by several baselines. In each case, we see improvements in test error
over baselines, even after we threshold fine-tuned stochastic pruning masks.
Finally, since a stochastic pruning mask induces a stochastic neural network,
we consider training the weights and/or pruning probabilities simultaneously to
minimize a PAC-Bayes bound on generalization error. Using data-dependent
priors, we obtain a selfbounded learning algorithm with strong performance and
numerically tight bounds. In the linear model, we show that a PAC-Bayes
generalization error bound is controlled by the magnitude of the change in
feature alignment between the 'prior' and 'posterior' data.
