On the Calibration of Nested Dichotomies for Large Multiclass Tasks
  Nested dichotomies are used as a method of transforming a multiclass
classification problem into a series of binary problems. A tree structure is
induced that recursively splits the set of classes into subsets, and a binary
classification model learns to discriminate between the two subsets of classes
at each node. In this paper, we demonstrate that these nested dichotomies
typically exhibit poor probability calibration, even when the base binary
models are well calibrated. We also show that this problem is exacerbated when
the binary models are poorly calibrated. We discuss the effectiveness of
different calibration strategies and show that accuracy and log-loss can be
significantly improved by calibrating both the internal base models and the
full nested dichotomy structure, especially when the number of classes is high.
