Which Samples Should be Learned First: Easy or Hard?
  An effective weighting scheme for training samples is essential for learning
tasks. Numerous weighting schemes have been proposed. Some schemes take the
easy-first mode, whereas some others take the hard-first one. Naturally, an
interesting yet realistic question is raised. Which samples should be learned
first given a new learning task, easy or hard? To answer this question, both
theoretical analyses and experimental verification are conducted. First, a
general optimized objective function is proposed, revealing the relationship
between the difficulty distribution and the difficulty-based sample weights.
Second, on the basis of the optimized objective function, theoretical answers
are obtained. Besides the easy-first and hard-first modes, there are two other
priority modes, namely, medium-first and two-ends-first. The prior mode does
not necessarily remain unchanged during the training process. Third, an
effective and universal solution is proposed to select the optimal priority
mode when there is no prior knowledge or theoretical clues. The four modes,
namely, easy/medium/hard/two-ends-first, can be flexibly switched in the
proposed solution. Fourth, a wide range of experiments is conducted under
various scenarios to further compare the weighting schemes in different modes.
On the basis of these works, reasonable and comprehensive answers are obtained.
Factors including the distribution of samples' learning difficulties and the
validation data determine which samples should be learned first in a learning
task.
