The Variational Bandwidth Bottleneck: Stochastic Evaluation on an
  Information Budget
  In many applications, it is desirable to extract only the relevant
information from complex input data, which involves making a decision about
which input features are relevant. The information bottleneck method formalizes
this as an information-theoretic optimization problem by maintaining an optimal
tradeoff between compression (throwing away irrelevant input information), and
predicting the target. In many problem settings, including the reinforcement
learning problems we consider in this work, we might prefer to compress only
part of the input. This is typically the case when we have a standard
conditioning input, such as a state observation, and a "privileged" input,
which might correspond to the goal of a task, the output of a costly planning
algorithm, or communication with another agent. In such cases, we might prefer
to compress the privileged input, either to achieve better generalization
(e.g., with respect to goals) or to minimize access to costly information
(e.g., in the case of communication). Practical implementations of the
information bottleneck based on variational inference require access to the
privileged input in order to compute the bottleneck variable, so although they
perform compression, this compression operation itself needs unrestricted,
lossless access. In this work, we propose the variational bandwidth bottleneck,
which decides for each example on the estimated value of the privileged
information before seeing it, i.e., only based on the standard input, and then
accordingly chooses stochastically, whether to access the privileged input or
not. We formulate a tractable approximation to this framework and demonstrate
in a series of reinforcement learning experiments that it can improve
generalization and reduce access to computationally costly information.
