Kernel Methods and Multi-layer Perceptrons Learn Linear Models in High
  Dimensions
  Empirical observation of high dimensional phenomena, such as the double
descent behaviour, has attracted a lot of interest in understanding classical
techniques such as kernel methods, and their implications to explain
generalization properties of neural networks. Many recent works analyze such
models in a certain high-dimensional regime where the covariates are
independent and the number of samples and the number of covariates grow at a
fixed ratio (i.e. proportional asymptotics). In this work we show that for a
large class of kernels, including the neural tangent kernel of fully connected
networks, kernel methods can only perform as well as linear models in this
regime. More surprisingly, when the data is generated by a kernel model where
the relationship between input and the response could be very nonlinear, we
show that linear models are in fact optimal, i.e. linear models achieve the
minimum risk among all models, linear or nonlinear. These results suggest that
more complex models for the data other than independent features are needed for
high-dimensional analysis.
