Learning from Mistakes based on Class Weighting with Application to
  Neural Architecture Search
  Learning from mistakes is an effective learning approach widely used in human
learning, where a learner pays greater focus on mistakes to circumvent them in
the future to improve the overall learning outcomes. In this work, we aim to
investigate how effectively we can leverage this exceptional learning ability
to improve machine learning models. We propose a simple and effective
multi-level optimization framework called learning from mistakes using class
weighting (LFM-CW), inspired by mistake-driven learning to train better machine
learning models. In this formulation, the primary objective is to train a model
to perform effectively on target tasks by using a re-weighting technique. We
learn the class weights by minimizing the validation loss of the model and
re-train the model with the synthetic data from the image generator weighted by
class-wise performance and real data. We apply our LFM-CW framework with
differential architecture search methods on image classification datasets such
as CIFAR and ImageNet, where the results show that our proposed strategy
achieves lower error rate than the baselines.
