"Dependency Bottleneck" in Auto-encoding Architectures: an Empirical
  Study
  Recent works investigated the generalization properties in deep neural
networks (DNNs) by studying the Information Bottleneck in DNNs. However, the
mea- surement of the mutual information (MI) is often inaccurate due to the
density estimation. To address this issue, we propose to measure the dependency
instead of MI between layers in DNNs. Specifically, we propose to use
Hilbert-Schmidt Independence Criterion (HSIC) as the dependency measure, which
can measure the dependence of two random variables without estimating
probability densities. Moreover, HSIC is a special case of the Squared-loss
Mutual Information (SMI). In the experiment, we empirically evaluate the
generalization property using HSIC in both the reconstruction and prediction
auto-encoding (AE) architectures.
