Meta-Learning with Hessian-Free Approach in Deep Neural Nets Training
  Meta-learning is a promising method to achieve efficient training method
towards deep neural net and has been attracting increases interests in recent
years. But most of the current methods are still not capable to train complex
neuron net model with long-time training process. In this paper, a novel
second-order meta-optimizer, named Meta-learning with Hessian-Free(MLHF)
approach, is proposed based on the Hessian-Free approach. Two recurrent neural
networks are established to generate the damping and the precondition matrix of
this Hessian-Free framework. A series of techniques to meta-train the MLHF
towards stable and reinforce the meta-training of this optimizer, including the
gradient calculation of $H$. Numerical experiments on deep convolution neural
nets, including CUDA-convnet and ResNet18(v2), with datasets of CIFAR10 and
ILSVRC2012, indicate that the MLHF shows good and continuous training
performance during the whole long-time training process, i.e., both the
rapid-decreasing early stage and the steadily-deceasing later stage, and so is
a promising meta-learning framework towards elevating the training efficiency
in real-world deep neural nets.
