A partition-based similarity for classification distributions
  Herein we define a measure of similarity between classification distributions
that is both principled from the perspective of statistical pattern recognition
and useful from the perspective of machine learning practitioners. In
particular, we propose a novel similarity on classification distributions,
dubbed task similarity, that quantifies how an optimally-transformed optimal
representation for a source distribution performs when applied to inference
related to a target distribution. The definition of task similarity allows for
natural definitions of adversarial and orthogonal distributions. We highlight
limiting properties of representations induced by (universally) consistent
decision rules and demonstrate in simulation that an empirical estimate of task
similarity is a function of the decision rule deployed for inference. We
demonstrate that for a given target distribution, both transfer efficiency and
semantic similarity of candidate source distributions correlate with empirical
task similarity.
