Deep linear neural networks with arbitrary loss: All local minima are
  global
  We consider deep linear networks with arbitrary convex differentiable loss.
We provide a short and elementary proof of the fact that all local minima are
global minima if the hidden layers are either 1) at least as wide as the input
layer, or 2) at least as wide as the output layer. This result is the strongest
possible in the following sense: If the loss is convex and Lipschitz but not
differentiable then deep linear networks can have sub-optimal local minima.
