Phase transitions in nonparametric regressions: a curse of exploiting
  higher degree smoothness assumptions in finite samples
  When the regression function belongs to the smooth classes consisting of
univariate functions with derivatives up to the $(\gamma+1)$th order bounded in
absolute values by a common constant everywhere or a.e., it is generally viewed
that exploiting higher degree smoothness assumption helps reduce the estimation
error. This paper shows that the minimax optimal mean integrated squared error
(MISE) rate increases in $\gamma$ when the sample size $n$ is small relative to
$\left(\gamma+1\right)^{2\gamma+3}$ (e.g.,
$\left(\gamma+1\right)^{2\gamma+3}=262144$ when $\gamma=3$), and decreases in
$\gamma$ when $n$ is large relative to $\left(\gamma+1\right)^{2\gamma+3}$. In
particular, this phase transition property is shown to be achieved by common
nonparametric procedures. Consider $\gamma_{1}$ and $\gamma_{2}$ such that
$\gamma_{1}<\gamma_{2}$, where the $(\gamma_{2}+1)$th degree smoothness class
is a subset of the $(\gamma_{1}+1)$th degree class. What is interesting about
our results is that they imply, if $n$ is small relative to
$\left(\gamma_{1}+1\right)^{2\gamma_{1}+3}$, the optimal rate achieved by the
estimator constrained to be in the smoother class is larger. In data sets with
fewer than hundreds-of-thousands observations, our results suggest that one
should not exploit beyond the third degree of smoothness. To some extent, our
results provide a theoretical basis for the widely adopted practical
recommendation given by Gelman and Imbens (2019). The building blocks of our
minimax optimality results are a set of metric entropy bounds we develop in
this paper for smooth function classes. Some of our bounds are original, and
some of them refine and/or generalize the ones in the literature.
