Bayesian deep learning with hierarchical prior: Predictions from limited
  and noisy data
  Datasets in engineering applications are often limited and contaminated,
mainly due to unavoidable measurement noise and signal distortion. Thus, using
conventional data-driven approaches to build a reliable discriminative model,
and further applying this identified surrogate to uncertainty analysis remains
to be very challenging. A deep learning approach is presented to provide
predictions based on limited and noisy data. To address noise perturbation, the
Bayesian learning method that naturally facilitates an automatic updating
mechanism is considered to quantify and propagate model uncertainties into
predictive quantities. Specifically, hierarchical Bayesian modeling (HBM) is
first adopted to describe model uncertainties, which allows the prior
assumption to be less subjective, while also makes the proposed surrogate more
robust. Next, the Bayesian inference is seamlessly integrated into the DL
framework, which in turn supports probabilistic programming by yielding a
probability distribution of the quantities of interest rather than their point
estimates. Variational inference (VI) is implemented for the posterior
distribution analysis where the intractable marginalization of the likelihood
function over parameter space is framed in an optimization format, and
stochastic gradient descent method is applied to solve this optimization
problem. Finally, Monte Carlo simulation is used to obtain an unbiased
estimator in the predictive phase of Bayesian inference, where the proposed
Bayesian deep learning (BDL) scheme is able to offer confidence bounds for the
output estimation by analyzing propagated uncertainties. The effectiveness of
Bayesian shrinkage is demonstrated in improving predictive performance using
contaminated data, and various examples are provided to illustrate concepts,
methodologies, and algorithms of this proposed BDL modeling technique.
