On the Relevance-Complexity Region of Scalable Information Bottleneck
  The Information Bottleneck method is a learning technique that seeks a right
balance between accuracy and generalization capability through a suitable
tradeoff between compression complexity, measured by minimum description
length, and distortion evaluated under logarithmic loss measure. In this paper,
we study a variation of the problem, called scalable information bottleneck,
where the encoder outputs multiple descriptions of the observation with
increasingly richer features. The problem at hand is motivated by some
application scenarios that require varying levels of accuracy depending on the
allowed level of generalization. First, we establish explicit (analytic)
characterizations of the relevance-complexity region for memoryless Gaussian
sources and memoryless binary sources. Then, we derive a Blahut-Arimoto type
algorithm that allows us to compute (an approximation of) the region for
general discrete sources. Finally, an application example in the pattern
classification problem is provided along with numerical results.
