Estimating the standard error of cross-Validation-Based estimators of
  classifier performance
  First, we analyze the variance of the Cross Validation (CV)-based estimators
used for estimating the performance of classification rules. Second, we propose
a novel estimator to estimate this variance using the Influence Function (IF)
approach that had been used previously very successfully to estimate the
variance of the bootstrap-based estimators. The motivation for this research is
that, as the best of our knowledge, the literature lacks a rigorous method for
estimating the variance of the CV-based estimators. What is available is a set
of ad-hoc procedures that have no mathematical foundation since they ignore the
covariance structure among dependent random variables. The conducted
experiments show that the IF proposed method has small RMS error with some
bias. However, surprisingly, the ad-hoc methods still work better than the
IF-based method. Unfortunately, this is due to the lack of enough smoothness if
compared to the bootstrap estimator. This opens the research for three points:
(1) more comprehensive simulation study to clarify when the IF method win or
loose; (2) more mathematical analysis to figure out why the ad-hoc methods work
well; and (3) more mathematical treatment to figure out the connection between
the appropriate amount of "smoothness" and decreasing the bias of the IF
method.
