Hessian Eigenspectra of More Realistic Nonlinear Models
  Given an optimization problem, the Hessian matrix and its eigenspectrum can
be used in many ways, ranging from designing more efficient second-order
algorithms to performing model analysis and regression diagnostics. When
nonlinear models and non-convex problems are considered, strong simplifying
assumptions are often made to make Hessian spectral analysis more tractable.
This leads to the question of how relevant the conclusions of such analyses are
for more realistic nonlinear models. In this paper, we exploit deterministic
equivalent techniques from random matrix theory to make a \emph{precise}
characterization of the Hessian eigenspectra for a broad family of nonlinear
models, including models that generalize the classical generalized linear
models, without relying on strong simplifying assumptions used previously. We
show that, depending on the data properties, the nonlinear response model, and
the loss function, the Hessian can have \emph{qualitatively} different spectral
behaviors: of bounded or unbounded support, with single- or multi-bulk, and
with isolated eigenvalues on the left- or right-hand side of the bulk. By
focusing on such a simple but nontrivial nonlinear model, our analysis takes a
step forward to unveil the theoretical origin of many visually striking
features observed in more complex machine learning models.
