Information-theoretic Classification Accuracy: A Criterion that Guides
  Data-driven Combination of Ambiguous Outcome Labels in Multi-class
  Classification
  Outcome labeling ambiguity and subjectivity are ubiquitous in real-world
datasets. While practitioners commonly combine ambiguous outcome labels in an
ad hoc way to improve the accuracy of multi-class classification, there lacks a
principled approach to guide label combination by any optimality criterion. To
address this problem, we propose the information-theoretic classification
accuracy (ITCA), a criterion of outcome "information" conditional on outcome
prediction, to guide practitioners on how to combine ambiguous outcome labels.
ITCA indicates a balance in the trade-off between prediction accuracy (how well
do predicted labels agree with actual labels) and prediction resolution (how
many labels are predictable). To find the optimal label combination indicated
by ITCA, we develop two search strategies: greedy search and breadth-first
search. Notably, ITCA and the two search strategies are adaptive to all
machine-learning classification algorithms. Coupled with a classification
algorithm and a search strategy, ITCA has two uses: to improve prediction
accuracy and to identify ambiguous labels. We first verify that ITCA achieves
high accuracy with both search strategies in finding the correct label
combinations on synthetic and real data. Then we demonstrate the effectiveness
of ITCA in diverse applications including medical prognosis, cancer survival
prediction, user demographics prediction, and cell type classification.
