Studying the Interplay between Information Loss and Operation Loss in
  Representations for Classification
  Information-theoretic measures have been widely adopted in the design of
features for learning and decision problems. Inspired by this, we look at the
relationship between i) a weak form of information loss in the Shannon sense
and ii) the operation loss in the minimum probability of error (MPE) sense when
considering a family of lossy continuous representations (features) of a
continuous observation. We present several results that shed light on this
interplay. Our first result offers a lower bound on a weak form of information
loss as a function of its respective operation loss when adopting a discrete
lossy representation (quantization) instead of the original raw observation.
From this, our main result shows that a specific form of vanishing information
loss (a weak notion of asymptotic informational sufficiency) implies a
vanishing MPE loss (or asymptotic operational sufficiency) when considering a
general family of lossy continuous representations. Our theoretical findings
support the observation that the selection of feature representations that
attempt to capture informational sufficiency is appropriate for learning, but
this selection is a rather conservative design principle if the intended goal
is achieving MPE in classification. Supporting this last point, and under some
structural conditions, we show that it is possible to adopt an alternative
notion of informational sufficiency (strictly weaker than pure sufficiency in
the mutual information sense) to achieve operational sufficiency in learning.
