Trap of Feature Diversity in the Learning of MLPs
  In this paper, we focus on a typical two-phase phenomenon in the learning of
multi-layer perceptrons (MLPs), and we aim to explain the reason for the
decrease of feature diversity in the first phase. Specifically, people find
that, in the training of MLPs, the training loss does not decrease
significantly until the second phase. To this end, we further explore the
reason why the diversity of features over different samples keeps decreasing in
the first phase, which hurts the optimization of MLPs. We explain such a
phenomenon in terms of the learning dynamics of MLPs. Furthermore, we
theoretically explain why four typical operations can alleviate the decrease of
the feature diversity.
