Model Validation Using Mutated Training Labels: An Exploratory Study
  We introduce an exploratory study on Mutation Validation (MV), a model
validation method using mutated training labels for supervised learning. MV
mutates training data labels, retrains the model against the mutated data, then
uses the metamorphic relation that captures the consequent training performance
changes to assess model fit. It does not use a validation set or test set. The
intuition underpinning MV is that overfitting models tend to fit noise in the
training data. We explore 8 different learning algorithms, 18 datasets, and 5
types of hyperparameter tuning tasks. Our results demonstrate that MV is
accurate in model selection: the model recommendation hit rate is 92\% for MV
and less than 60\% for out-of-sample-validation. MV also provides more stable
hyperparameter tuning results than out-of-sample-validation across different
runs.
