Quasi-Newton Methods: A New Direction
  Four decades after their invention, quasi-Newton methods are still state of
the art in unconstrained numerical optimization. Although not usually
interpreted thus, these are learning algorithms that fit a local quadratic
approximation to the objective function. We show that many, including the most
popular, quasi-Newton methods can be interpreted as approximations of Bayesian
linear regression under varying prior assumptions. This new notion elucidates
some shortcomings of classical algorithms, and lights the way to a novel
nonparametric quasi-Newton method, which is able to make more efficient use of
available information at computational cost similar to its predecessors.
