Adaptive Network Sparsification with Dependent Variational
  Beta-Bernoulli Dropout
  While variational dropout approaches have been shown to be effective for
network sparsification, they are still suboptimal in the sense that they set
the dropout rate for each neuron without consideration of the input data. With
such input-independent dropout, each neuron is evolved to be generic across
inputs, which makes it difficult to sparsify networks without accuracy loss. To
overcome this limitation, we propose adaptive variational dropout whose
probabilities are drawn from sparsity-inducing beta Bernoulli prior. It allows
each neuron to be evolved either to be generic or specific for certain inputs,
or dropped altogether. Such input-adaptive sparsity-inducing dropout allows the
resulting network to tolerate larger degree of sparsity without losing its
expressive power by removing redundancies among features. We validate our
dependent variational beta-Bernoulli dropout on multiple public datasets, on
which it obtains significantly more compact networks than baseline methods,
with consistent accuracy improvements over the base networks.
