Informative Bayesian Neural Network Priors for Weak Signals
  Encoding domain knowledge into the prior over the high-dimensional weight
space of a neural network is challenging but essential in applications with
limited data and weak signals. Two types of domain knowledge are commonly
available in scientific applications: 1. feature sparsity (fraction of features
deemed relevant); 2. signal-to-noise ratio, quantified, for instance, as the
proportion of variance explained (PVE). We show how to encode both types of
domain knowledge into the widely used Gaussian scale mixture priors with
Automatic Relevance Determination. Specifically, we propose a new joint prior
over the local (i.e., feature-specific) scale parameters that encodes knowledge
about feature sparsity, and a Stein gradient optimization to tune the
hyperparameters in such a way that the distribution induced on the model's PVE
matches the prior distribution. We show empirically that the new prior improves
prediction accuracy, compared to existing neural network priors, on several
publicly available datasets and in a genetics application where signals are
weak and sparse, often outperforming even computationally intensive
cross-validation for hyperparameter tuning.
