Last Layer Marginal Likelihood for Invariance Learning
  Data augmentation is often used to incorporate inductive biases into models.
Traditionally, these are hand-crafted and tuned with cross validation. The
Bayesian paradigm for model selection provides a path towards end-to-end
learning of invariances using only the training data, by optimising the
marginal likelihood. Computing the marginal likelihood is hard for neural
networks, but success with tractable approaches that compute the marginal
likelihood for the last layer only raises the question of whether this
convenient approach might be employed for learning invariances. We show partial
success on standard benchmarks, in the low-data regime and on a medical imaging
dataset by designing a custom optimisation routine. Introducing a new lower
bound to the marginal likelihood allows us to perform inference for a larger
class of likelihood functions than before. On the other hand, we demonstrate
failure modes on the CIFAR10 dataset, where the last layer approximation is not
sufficient due to the increased complexity of our neural network. Our results
indicate that once more sophisticated approximations become available the
marginal likelihood is a promising approach for invariance learning in neural
networks.
