Disentangled Information Bottleneck
  The information bottleneck (IB) method is a technique for extracting
information that is relevant for predicting the target random variable from the
source random variable, which is typically implemented by optimizing the IB
Lagrangian that balances the compression and prediction terms. However, the IB
Lagrangian is hard to optimize, and multiple trials for tuning values of
Lagrangian multiplier are required. Moreover, we show that the prediction
performance strictly decreases as the compression gets stronger during
optimizing the IB Lagrangian. In this paper, we implement the IB method from
the perspective of supervised disentangling. Specifically, we introduce
Disentangled Information Bottleneck (DisenIB) that is consistent on compressing
source maximally without target prediction performance loss (maximum
compression). Theoretical and experimental results demonstrate that our method
is consistent on maximum compression, and performs well in terms of
generalization, robustness to adversarial attack, out-of-distribution
detection, and supervised disentangling.
