Entropic Issues in Likelihood-Based OOD Detection
  Deep generative models trained by maximum likelihood remain very popular
methods for reasoning about data probabilistically. However, it has been
observed that they can assign higher likelihoods to out-of-distribution (OOD)
data than in-distribution data, thus calling into question the meaning of these
likelihood values. In this work we provide a novel perspective on this
phenomenon, decomposing the average likelihood into a KL divergence term and an
entropy term. We argue that the latter can explain the curious OOD behaviour
mentioned above, suppressing likelihood values on datasets with higher entropy.
Although our idea is simple, we have not seen it explored yet in the
literature. This analysis provides further explanation for the success of OOD
detection methods based on likelihood ratios, as the problematic entropy term
cancels out in expectation. Finally, we discuss how this observation relates to
recent success in OOD detection with manifold-supported models, for which the
above decomposition does not hold.
