Optimization for deep learning: theory and algorithms
  When and why can a neural network be successfully trained? This article
provides an overview of optimization algorithms and theory for training neural
networks. First, we discuss the issue of gradient explosion/vanishing and the
more general issue of undesirable spectrum, and then discuss practical
solutions including careful initialization and normalization methods. Second,
we review generic optimization methods used in training neural networks, such
as SGD, adaptive gradient methods and distributed methods, and theoretical
results for these algorithms. Third, we review existing research on the global
issues of neural network training, including results on bad local minima, mode
connectivity, lottery ticket hypothesis and infinite-width analysis.
