Subject Cross Validation in Human Activity Recognition
  K-fold Cross Validation is commonly used to evaluate classifiers and tune
their hyperparameters. However, it assumes that data points are Independent and
Identically Distributed (i.i.d.) so that samples used in the training and test
sets can be selected randomly and uniformly. In Human Activity Recognition
datasets, we note that the samples produced by the same subjects are likely to
be correlated due to diverse factors. Hence, k-fold cross validation may
overestimate the performance of activity recognizers, in particular when
overlapping sliding windows are used. In this paper, we investigate the effect
of Subject Cross Validation on the performance of Human Activity Recognition,
both with non-overlapping and with overlapping sliding windows. Results show
that k-fold cross validation artificially increases the performance of
recognizers by about 10%, and even by 16% when overlapping windows are used. In
addition, we do not observe any performance gain from the use of overlapping
windows. We conclude that Human Activity Recognition systems should be
evaluated by Subject Cross Validation, and that overlapping windows are not
worth their extra computational cost.
