Exploratory Machine Learning with Unknown Unknowns
  In conventional supervised learning, a training dataset is given with
ground-truth labels from a known label set, and the learned model will classify
unseen instances to the known labels. In this paper, we study a new problem
setting in which there are unknown classes in the training dataset misperceived
as other labels, and thus their existence appears unknown from the given
supervision. We attribute the unknown unknowns to the fact that the training
dataset is badly advised by the incompletely perceived label space due to the
insufficient feature information. To this end, we propose the exploratory
machine learning, which examines and investigates the training dataset by
actively augmenting the feature space to discover potentially unknown labels.
Our approach consists of three ingredients including rejection model, feature
acquisition, and model cascade. The effectiveness is validated on both
synthetic and real datasets.
