Effective Regularization Through Loss-Function Metalearning
  Evolutionary optimization, such as the TaylorGLO method, can be used to
discover novel, customized loss functions for deep neural networks, resulting
in improved performance, faster training, and improved data utilization. A
likely explanation is that such functions discourage overfitting, leading to
effective regularization. This paper demonstrates theoretically that this is
indeed the case for TaylorGLO: Decomposition of learning rules makes it
possible to characterize the training dynamics and show that the loss functions
evolved by TaylorGLO balance the pull to zero error, and a push away from it to
avoid overfitting. They may also automatically take advantage of label
smoothing. This analysis leads to an invariant that can be utilized to make the
metalearning process more efficient in practice; the mechanism also results in
networks that are robust against adversarial attacks. Loss-function evolution
can thus be seen as a well-founded new aspect of metalearning in neural
networks.
