Failures of Gradient-Based Deep Learning
  In recent years, Deep Learning has become the go-to solution for a broad
range of applications, often outperforming state-of-the-art. However, it is
important, for both theoreticians and practitioners, to gain a deeper
understanding of the difficulties and limitations associated with common
approaches and algorithms. We describe four types of simple problems, for which
the gradient-based algorithms commonly used in deep learning either fail or
suffer from significant difficulties. We illustrate the failures through
practical experiments, and provide theoretical insights explaining their
source, and how they might be remedied.
