Quasi-Newton's method in the class gradient defined high-curvature
  subspace
  Classification problems using deep learning have been shown to have a
high-curvature subspace in the loss landscape equal in dimension to the number
of classes. Moreover, this subspace corresponds to the subspace spanned by the
logit gradients for each class. An obvious strategy to speed up optimisation
would be to use Newton's method in the high-curvature subspace and stochastic
gradient descent in the co-space. We show that a naive implementation actually
slows down convergence and we speculate why this might be.
