Rethinking Exponential Averaging of the Fisher
  In optimization for Machine learning (ML), it is typical that
curvature-matrix (CM) estimates rely on an exponential average (EA) of local
estimates (giving EA-CM algorithms). This approach has little principled
justification, but is very often used in practice. In this paper, we draw a
connection between EA-CM algorithms and what we call a "Wake of Quadratic
regularized models". The outlined connection allows us to understand what EA-CM
algorithms are doing from an optimization perspective. Generalizing from the
established connection, we propose a new family of algorithms, "KL-Divergence
Wake-Regularized Models" (KLD-WRM). We give three different practical
instantiations of KLD-WRM, and show numerical results where we outperform
K-FAC.
