Data augmentation in Bayesian neural networks and the cold posterior
  effect
  Bayesian neural networks that incorporate data augmentation implicitly use a
``randomly perturbed log-likelihood [which] does not have a clean
interpretation as a valid likelihood function'' (Izmailov et al. 2021). Here,
we provide several approaches to developing principled Bayesian neural networks
incorporating data augmentation. We introduce a ``finite orbit'' setting which
allows likelihoods to be computed exactly, and give tight multi-sample bounds
in the more usual ``full orbit'' setting. These models cast light on the origin
of the cold posterior effect. In particular, we find that the cold posterior
effect persists even in these principled models incorporating data
augmentation. This suggests that the cold posterior effect cannot be dismissed
as an artifact of data augmentation using incorrect likelihoods.
