On the Estimation of Derivatives Using Plug-in KRR Estimators
  We study the problem of estimating the derivatives of the regression
function, which has a wide range of applications as a key nonparametric
functional of unknown functions. Standard analysis may be tailored to specific
derivative orders, and parameter tuning remains a daunting challenge
particularly for high-order derivatives. In this article, we propose a simple
plug-in kernel ridge regression (KRR) estimator in nonparametric regression
with random design that is broadly applicable for multi-dimensional support and
arbitrary mixed-partial derivatives. We provide a non-asymptotic analysis to
study the behavior of the proposed estimator, leading to two error bounds for a
general class of kernels under the strong $L_\infty$ norm. In a concrete
example specialized to kernels with polynomially decaying eigenvalues, the
proposed estimator recovers the minimax optimal rate up to a logarithmic factor
for estimating derivatives of functions in H\"older class. Interestingly, the
proposed estimator achieves the optimal rate of convergence with the same
choice of tuning parameter for any order of derivatives. Hence, the proposed
estimator enjoys a remarkable \textit{plug-in property} for derivatives in that
it automatically adapts to the order of derivatives to be estimated, enabling
easy tuning in practice. Our simulation studies show favorable finite sample
performance of the proposed method relative to several existing methods.
