Continuity of Generalized Entropy and Statistical Learning
  We study the continuity property of the generalized entropy as a function of
the underlying probability distribution, defined with an action space and a
loss function, and use this property to answer the basic questions in
statistical learning theory: the excess risk analyses for various learning
methods. We first derive upper and lower bounds for the entropy difference of
two distributions in terms of several commonly used f-divergences, the
Wasserstein distance, a distance that depends on the action space and the loss
function, and the Bregman divergence generated by the entropy, which also
induces bounds in terms of the Euclidean distance between the two
distributions. Examples are given along with the discussion of each general
result, comparisons are made with the existing entropy difference bounds, and
new mutual information upper bounds are derived based on the new results. We
then apply the entropy difference bounds to the theory of statistical learning.
It is shown that the excess risks in the two popular learning paradigms, the
frequentist learning and the Bayesian learning, both can be studied with the
continuity property of different forms of the generalized entropy. The analysis
is then extended to the continuity of generalized conditional entropy. The
extension provides performance bounds for Bayes decision making with mismatched
distributions. It also leads to excess risk bounds for a third paradigm of
learning, where the decision rule is optimally designed under the projection of
the empirical distribution to a predefined family of distributions. We thus
establish a unified method of excess risk analysis for the three major
paradigms of statistical learning, through the continuity of generalized
entropy.
