Plateau Phenomenon in Gradient Descent Training of ReLU networks:
  Explanation, Quantification and Avoidance
  The ability of neural networks to provide `best in class' approximation
across a wide range of applications is well-documented. Nevertheless, the
powerful expressivity of neural networks comes to naught if one is unable to
effectively train (choose) the parameters defining the network. In general,
neural networks are trained by gradient descent type optimization methods, or a
stochastic variant thereof. In practice, such methods result in the loss
function decreases rapidly at the beginning of training but then, after a
relatively small number of steps, significantly slow down. The loss may even
appear to stagnate over the period of a large number of epochs, only to then
suddenly start to decrease fast again for no apparent reason. This so-called
plateau phenomenon manifests itself in many learning tasks.
  The present work aims to identify and quantify the root causes of plateau
phenomenon. No assumptions are made on the number of neurons relative to the
number of training data, and our results hold for both the lazy and adaptive
regimes. The main findings are: plateaux correspond to periods during which
activation patterns remain constant, where activation pattern refers to the
number of data points that activate a given neuron; quantification of
convergence of the gradient flow dynamics; and, characterization of stationary
points in terms solutions of local least squares regression lines over subsets
of the training data. Based on these conclusions, we propose a new iterative
training method, the Active Neuron Least Squares (ANLS), characterised by the
explicit adjustment of the activation pattern at each step, which is designed
to enable a quick exit from a plateau. Illustrative numerical examples are
included throughout.
