When Do Curricula Work?
  Inspired by human learning, researchers have proposed ordering examples
during training based on their difficulty. Both curriculum learning, exposing a
network to easier examples early in training, and anti-curriculum learning,
showing the most difficult examples first, have been suggested as improvements
to the standard i.i.d. training. In this work, we set out to investigate the
relative benefits of ordered learning. We first investigate the \emph{implicit
curricula} resulting from architectural and optimization bias and find that
samples are learned in a highly consistent order. Next, to quantify the benefit
of \emph{explicit curricula}, we conduct extensive experiments over thousands
of orderings spanning three kinds of learning: curriculum, anti-curriculum, and
random-curriculum -- in which the size of the training dataset is dynamically
increased over time, but the examples are randomly ordered. We find that for
standard benchmark datasets, curricula have only marginal benefits, and that
randomly ordered samples perform as well or better than curricula and
anti-curricula, suggesting that any benefit is entirely due to the dynamic
training set size. Inspired by common use cases of curriculum learning in
practice, we investigate the role of limited training time budget and noisy
data in the success of curriculum learning. Our experiments demonstrate that
curriculum, but not anti-curriculum can indeed improve the performance either
with limited training time budget or in existence of noisy data.
