Spatial-Temporal-Fusion BNN: Variational Bayesian Feature Layer
  Bayesian neural networks (BNNs) have become a principal approach to alleviate
overconfident predictions in deep learning, but they often suffer from scaling
issues due to a large number of distribution parameters. In this paper, we
discover that the first layer of a deep network possesses multiple disparate
optima when solely retrained. This indicates a large posterior variance when
the first layer is altered by a Bayesian layer, which motivates us to design a
spatial-temporal-fusion BNN (STF-BNN) for efficiently scaling BNNs to large
models: (1) first normally train a neural network from scratch to realize fast
training; and (2) the first layer is converted to Bayesian and inferred by
employing stochastic variational inference, while other layers are fixed.
Compared to vanilla BNNs, our approach can greatly reduce the training time and
the number of parameters, which contributes to scale BNNs efficiently. We
further provide theoretical guarantees on the generalizability and the
capability of mitigating overconfidence of STF-BNN. Comprehensive experiments
demonstrate that STF-BNN (1) achieves the state-of-the-art performance on
prediction and uncertainty quantification; (2) significantly improves
adversarial robustness and privacy preservation; and (3) considerably reduces
training time and memory costs.
