A finite sample analysis of the benign overfitting phenomenon for ridge
  function estimation
  Recent extensive numerical experiments in high scale machine learning have
allowed to uncover a quite counterintuitive phase transition, as a function of
the ratio between the sample size and the number of parameters in the model. As
the number of parameters $p$ approaches the sample size $n$, the generalisation
error (a.k.a. testing error) increases, but in many cases, it starts decreasing
again past the threshold $p=n$. This surprising phenomenon, brought to the
theoretical community attention in \cite{belkin2019reconciling}, has been
thoroughly investigated lately, more specifically for simpler models than deep
neural networks, such as the linear model when the parameter is taken to be the
minimum norm solution to the least-square problem, mostly in the asymptotic
regime when $p$ and $n$ tend to $+\infty$; see e.g. \cite{hastie2019surprises}.
In the present paper, we propose a finite sample analysis of non-linear models
of \textit{ridge} type, where we investigate the \textit{overparametrised
regime} of the double descent phenomenon for both the \textit{estimation
problem} and the \textit{prediction} problem. Our results provide a precise
analysis of the distance of the best estimator from the true parameter as well
as a generalisation bound which complements recent works of
\cite{bartlett2020benign} and \cite{chinot2020benign}. Our analysis is based on
efficient but elementary tools closely related to the continuous Newton method
\cite{neuberger2007continuous}.
