Sparsity-Inducing Categorical Prior Improves Robustness of the
  Information Bottleneck
  The information bottleneck framework provides a systematic approach to learn
representations that compress nuisance information in inputs and extract
semantically meaningful information about the predictions. However, the choice
of the prior distribution that fix the dimensionality across all the data can
restrict the flexibility of this approach to learn robust representations. We
present a novel sparsity-inducing spike-slab prior that uses sparsity as a
mechanism to provide flexibility that allows each data point to learn its own
dimension distribution. In addition, it provides a mechanism to learn a joint
distribution of the latent variable and the sparsity. Thus, unlike other
approaches, it can account for the full uncertainty in the latent space.
Through a series of experiments using in-distribution and out-of-distribution
learning scenarios on the MNIST and Fashion-MNIST data we show that the
proposed approach improves the accuracy and robustness compared with the
traditional fixed -imensional priors as well as other sparsity-induction
mechanisms proposed in the literature.
