Adversarial Training Reduces Information and Improves Transferability
  Recent results show that features of adversarially trained networks for
classification, in addition to being robust, enable desirable properties such
as invertibility. The latter property may seem counter-intuitive as it is
widely accepted by the community that classification models should only capture
the minimal information (features) required for the task. Motivated by this
discrepancy, we investigate the dual relationship between Adversarial Training
and Information Theory. We show that the Adversarial Training can improve
linear transferability to new tasks, from which arises a new trade-off between
transferability of representations and accuracy on the source task. We validate
our results employing robust networks trained on CIFAR-10, CIFAR-100 and
ImageNet on several datasets. Moreover, we show that Adversarial Training
reduces Fisher information of representations about the input and of the
weights about the task, and we provide a theoretical argument which explains
the invertibility of deterministic networks without violating the principle of
minimality. Finally, we leverage our theoretical insights to remarkably improve
the quality of reconstructed images through inversion.
