Data Augmentation for Bayesian Deep Learning
  Deep Learning (DL) methods have emerged as one of the most powerful tools for
functional approximation and prediction. While the representation properties of
DL have been well studied, uncertainty quantification remains challenging and
largely unexplored. Data augmentation techniques are a natural approach to
provide uncertainty quantification and to incorporate stochastic Monte Carlo
search into stochastic gradient descent (SGD) methods. The purpose of our paper
is to show that training DL architectures with data augmentation leads to
efficiency gains. We use the theory of scale mixtures of normals to derive data
augmentation strategies for deep learning. This allows variants of the
expectation-maximization and MCMC algorithms to be brought to bear on these
high dimensional nonlinear deep learning models. To demonstrate our
methodology, we develop data augmentation algorithms for a variety of commonly
used activation functions: logit, ReLU, leaky ReLU and SVM. Our methodology is
compared to traditional stochastic gradient descent with back-propagation. Our
optimization procedure leads to a version of iteratively re-weighted least
squares and can be implemented at scale with accelerated linear algebra methods
providing substantial improvement in speed. We illustrate our methodology on a
number of standard datasets. Finally, we conclude with directions for future
research.
