Pre-Train Your Loss: Easy Bayesian Transfer Learning with Informative
  Priors
  Deep learning is increasingly moving towards a transfer learning paradigm
whereby large foundation models are fine-tuned on downstream tasks, starting
from an initialization learned on the source task. But an initialization
contains relatively little information about the source task. Instead, we show
that we can learn highly informative posteriors from the source task, through
supervised or self-supervised approaches, which then serve as the basis for
priors that modify the whole loss surface on the downstream task. This simple
modular approach enables significant performance gains and more data-efficient
learning on a variety of downstream classification and segmentation tasks,
serving as a drop-in replacement for standard pre-training strategies. These
highly informative priors also can be saved for future use, similar to
pre-trained weights, and stand in contrast to the zero-mean isotropic
uninformative priors that are typically used in Bayesian deep learning.
