Fluctuation-dissipation relations for stochastic gradient descent
  The notion of the stationary equilibrium ensemble has played a central role
in statistical mechanics. In machine learning as well, training serves as
generalized equilibration that drives the probability distribution of model
parameters toward stationarity. Here, we derive stationary
fluctuation-dissipation relations that link measurable quantities and
hyperparameters in the stochastic gradient descent algorithm. These relations
hold exactly for any stationary state and can in particular be used to
adaptively set training schedule. We can further use the relations to
efficiently extract information pertaining to a loss-function landscape such as
the magnitudes of its Hessian and anharmonicity. Our claims are empirically
verified.
