Understanding Memorization from the Perspective of Optimization via
  Efficient Influence Estimation
  Over-parameterized deep neural networks are able to achieve excellent
training accuracy while maintaining a small generalization error. It has also
been found that they are able to fit arbitrary labels, and this behaviour is
referred to as the phenomenon of memorization. In this work, we study the
phenomenon of memorization with turn-over dropout, an efficient method to
estimate influence and memorization, for data with true labels (real data) and
data with random labels (random data). Our main findings are: (i) For both real
data and random data, the optimization of easy examples (e.g., real data) and
difficult examples (e.g., random data) are conducted by the network
simultaneously, with easy ones at a higher speed; (ii) For real data, a correct
difficult example in the training dataset is more informative than an easy one.
By showing the existence of memorization on random data and real data, we
highlight the consistency between them regarding optimization and we emphasize
the implication of memorization during optimization.
