A critical look at the current train/test split in machine learning
  The randomized or cross-validated split of training and testing sets has been
adopted as the gold standard of machine learning for decades. The establishment
of these split protocols are based on two assumptions: (i)-fixing the dataset
to be eternally static so we could evaluate different machine learning
algorithms or models; (ii)-there is a complete set of annotated data available
to researchers or industrial practitioners. However, in this article, we intend
to take a closer and critical look at the split protocol itself and point out
its weakness and limitation, especially for industrial applications. In many
real-world problems, we must acknowledge that there are numerous situations
where assumption (ii) does not hold. For instance, for interdisciplinary
applications like drug discovery, it often requires real lab experiments to
annotate data which poses huge costs in both time and financial considerations.
In other words, it can be very difficult or even impossible to satisfy
assumption (ii). In this article, we intend to access this problem and
reiterate the paradigm of active learning, and investigate its potential on
solving problems under unconventional train/test split protocols. We further
propose a new adaptive active learning architecture (AAL) which involves an
adaptation policy, in comparison with the traditional active learning that only
unidirectionally adds data points to the training pool. We primarily justify
our points by extensively investigating an interdisciplinary drug-protein
binding problem. We additionally evaluate AAL on more conventional machine
learning benchmarking datasets like CIFAR-10 to demonstrate the
generalizability and efficacy of the new framework.
