On the Limits of Learning Representations with Label-Based Supervision
  Advances in neural network based classifiers have transformed automatic
feature learning from a pipe dream of stronger AI to a routine and expected
property of practical systems. Since the emergence of AlexNet every winning
submission of the ImageNet challenge has employed end-to-end representation
learning, and due to the utility of good representations for transfer learning,
representation learning has become as an important and distinct task from
supervised learning. At present, this distinction is inconsequential, as
supervised methods are state-of-the-art in learning transferable
representations. But recent work has shown that generative models can also be
powerful agents of representation learning. Will the representations learned
from these generative methods ever rival the quality of those from their
supervised competitors? In this work, we argue in the affirmative, that from an
information theoretic perspective, generative models have greater potential for
representation learning. Based on several experimentally validated assumptions,
we show that supervised learning is upper bounded in its capacity for
representation learning in ways that certain generative models, such as
Generative Adversarial Networks (GANs) are not. We hope that our analysis will
provide a rigorous motivation for further exploration of generative
representation learning.
