An iterative K-FAC algorithm for Deep Learning
  Kronecker-factored Approximate Curvature (K-FAC) method is a high efficiency
second order optimizer for the deep learning. Its training time is less than
SGD(or other first-order method) with same accuracy in many large-scale
problems. The key of K-FAC is to approximates Fisher information matrix (FIM)
as a block-diagonal matrix where each block is an inverse of tiny Kronecker
factors. In this short note, we present CG-FAC -- an new iterative K-FAC
algorithm. It uses conjugate gradient method to approximate the nature
gradient. This CG-FAC method is matrix-free, that is, no need to generate the
FIM matrix, also no need to generate the Kronecker factors A and G. We prove
that the time and memory complexity of iterative CG-FAC is much less than that
of standard K-FAC algorithm.
