Machine Learning Assessment: implications to cybersecurity
  This chapter is dedicated to the assessment and performance estimation of
machine learning (ML) algorithms, a topic that is equally important to the
construction of these algorithms, in particular in the context of cyberphysical
security design. The literature is full of nonparametric methods to estimate a
statistic from just one available dataset through resampling techniques, e.g.,
jackknife, bootstrap and cross validation (CV). Special statistics of great
interest are the error rate and the area under the ROC curve (AUC) of a
classification rule. The importance of these resampling methods stems from the
fact that they require no knowledge about the probability distribution of the
data or the construction details of the ML algorithm. This chapter provides a
concise review of this literature to establish a coherent theoretical framework
for these methods that can estimate both the error rate (a one-sample
statistic) and the AUC (a two-sample statistic). The resampling methods are
usually computationally expensive, because they rely on repeating the training
and testing of a ML algorithm after each resampling iteration. Therefore, the
practical applicability of some of these methods may be limited to the
traditional ML algorithms rather than the very computationally demanding
approaches of the recent deep neural networks (DNN). In the field of
cyberphysical security, many applications generate structured (tabular) data,
which can be fed to all traditional ML approaches. This is in contrast to the
DNN approaches, which favor unstructured data, e.g., images, text, voice, etc.;
hence, the relevance of this chapter to this field.%
