The Variational Deficiency Bottleneck
  We introduce a bottleneck method for learning data representations based on
information deficiency, rather than the more traditional information
sufficiency. A variational upper bound allows us to implement this method
efficiently. The bound itself is bounded above by the variational information
bottleneck objective, and the two methods coincide in the regime of single-shot
Monte Carlo approximations. The notion of deficiency provides a principled way
of approximating complicated channels by relatively simpler ones. We show that
the deficiency of one channel with respect to another has an operational
interpretation in terms of the optimal risk gap of decision problems, capturing
classification as a special case. Experiments demonstrate that the deficiency
bottleneck can provide advantages in terms of minimal sufficiency as measured
by information bottleneck curves, while retaining robust test performance in
classification tasks.
