A Free-Energy Principle for Representation Learning
  This paper employs a formal connection of machine learning with
thermodynamics to characterize the quality of learnt representations for
transfer learning. We discuss how information-theoretic functional such as
rate, distortion and classification loss of a model lie on a convex, so-called
equilibrium surface.We prescribe dynamical processes to traverse this surface
under constraints, e.g., an iso-classification process that trades off rate and
distortion to keep the classification loss unchanged. We demonstrate how this
process can be used for transferring representations from a source dataset to a
target dataset while keeping the classification loss constant. Experimental
validation of the theoretical results is provided on standard
image-classification datasets.
