Toward a Characterization of Loss Functions for Distribution Learning
  In this work we study loss functions for learning and evaluating probability
distributions over large discrete domains. Unlike classification or regression
where a wide variety of loss functions are used, in the distribution learning
and density estimation literature, very few losses outside the dominant $log\
loss$ are applied. We aim to understand this fact, taking an axiomatic approach
to the design of loss functions for learning distributions. We start by
proposing a set of desirable criteria that any good loss function should
satisfy. Intuitively, these criteria require that the loss function faithfully
evaluates a candidate distribution, both in expectation and when estimated on a
few samples. Interestingly, we observe that \emph{no loss function} possesses
all of these criteria. However, one can circumvent this issue by introducing a
natural restriction on the set of candidate distributions. Specifically, we
require that candidates are $calibrated$ with respect to the target
distribution, i.e., they may contain less information than the target but
otherwise do not significantly distort the truth. We show that, after
restricting to this set of distributions, the log loss, along with a large
variety of other losses satisfy the desired criteria. These results pave the
way for future investigations of distribution learning that look beyond the log
loss, choosing a loss function based on application or domain need.
