Gradients are Not All You Need
  Differentiable programming techniques are widely used in the community and
are responsible for the machine learning renaissance of the past several
decades. While these methods are powerful, they have limits. In this short
report, we discuss a common chaos based failure mode which appears in a variety
of differentiable circumstances, ranging from recurrent neural networks and
numerical physics simulation to training learned optimizers. We trace this
failure to the spectrum of the Jacobian of the system under study, and provide
criteria for when a practitioner might expect this failure to spoil their
differentiation based optimization algorithms.
