Nonequilibrium thermodynamics of self-supervised learning
  Self-supervised learning (SSL) of energy based models has an intuitive
relation to equilibrium thermodynamics because the softmax layer, mapping
energies to probabilities, is a Gibbs distribution. However, in what way SSL is
a thermodynamic process? We show that some SSL paradigms behave as a
thermodynamic composite system formed by representations and self-labels in
contact with a nonequilibrium reservoir. Moreover, this system is subjected to
usual thermodynamic cycles, such as adiabatic expansion and isochoric heating,
resulting in a generalized Gibbs ensemble (GGE). In this picture, we show that
learning is seen as a demon that operates in cycles using feedback measurements
to extract negative work from the system. As applications, we examine some SSL
algorithms using this idea.
