Depth induces scale-averaging in overparameterized linear Bayesian
  neural networks
  Inference in deep Bayesian neural networks is only fully understood in the
infinite-width limit, where the posterior flexibility afforded by increased
depth washes out and the posterior predictive collapses to a shallow Gaussian
process. Here, we interpret finite deep linear Bayesian neural networks as
data-dependent scale mixtures of Gaussian process predictors across output
channels. We leverage this observation to study representation learning in
these networks, allowing us to connect limiting results obtained in previous
studies within a unified framework. In total, these results advance our
analytical understanding of how depth affects inference in a simple class of
Bayesian neural networks.
