Overly Optimistic Prediction Results on Imbalanced Data: a Case Study of
  Flaws and Benefits when Applying Over-sampling
  Information extracted from electrohysterography recordings could potentially
prove to be an interesting additional source of information to estimate the
risk on preterm birth. Recently, a large number of studies have reported
near-perfect results to distinguish between recordings of patients that will
deliver term or preterm using a public resource, called the Term/Preterm
Electrohysterogram database. However, we argue that these results are overly
optimistic due to a methodological flaw being made. In this work, we focus on
one specific type of methodological flaw: applying over-sampling before
partitioning the data into mutually exclusive training and testing sets. We
show how this causes the results to be biased using two artificial datasets and
reproduce results of studies in which this flaw was identified. Moreover, we
evaluate the actual impact of over-sampling on predictive performance, when
applied prior to data partitioning, using the same methodologies of related
studies, to provide a realistic view of these methodologies' generalization
capabilities. We make our research reproducible by providing all the code under
an open license.
