Is the Meta-Learning Idea Able to Improve the Generalization of Deep
  Neural Networks on the Standard Supervised Learning?
  Substantial efforts have been made on improving the generalization abilities
of deep neural networks (DNNs) in order to obtain better performances without
introducing more parameters. On the other hand, meta-learning approaches
exhibit powerful generalization on new tasks in few-shot learning. Intuitively,
few-shot learning is more challenging than the standard supervised learning as
each target class only has a very few or no training samples. The natural
question that arises is whether the meta-learning idea can be used for
improving the generalization of DNNs on the standard supervised learning. In
this paper, we propose a novel meta-learning based training procedure (MLTP)
for DNNs and demonstrate that the meta-learning idea can indeed improve the
generalization abilities of DNNs. MLTP simulates the meta-training process by
considering a batch of training samples as a task. The key idea is that the
gradient descent step for improving the current task performance should also
improve a new task performance, which is ignored by the current standard
procedure for training neural networks. MLTP also benefits from all the
existing training techniques such as dropout, weight decay, and batch
normalization. We evaluate MLTP by training a variety of small and large neural
networks on three benchmark datasets, i.e., CIFAR-10, CIFAR-100, and Tiny
ImageNet. The experimental results show a consistently improved generalization
performance on all the DNNs with different sizes, which verifies the promise of
MLTP and demonstrates that the meta-learning idea is indeed able to improve the
generalization of DNNs on the standard supervised learning.
