Sparse Bayesian Deep Learning for Dynamic System Identification
  This paper proposes a sparse Bayesian treatment of deep neural networks
(DNNs) for system identification. Although DNNs show impressive approximation
ability in various fields, several challenges still exist for system
identification problems. First, DNNs are known to be too complex that they can
easily overfit the training data. Second, the selection of the input regressors
for system identification is nontrivial. Third, uncertainty quantification of
the model parameters and predictions are necessary. The proposed Bayesian
approach offers a principled way to alleviate the above challenges by marginal
likelihood/model evidence approximation and structured group sparsity-inducing
priors construction. The identification algorithm is derived as an iterative
regularised optimisation procedure that can be solved as efficiently as
training typical DNNs. Remarkably, an efficient and recursive Hessian
calculation method for each layer of DNNs is developed, turning the intractable
training/optimisation process into a tractable one. Furthermore, a practical
calculation approach based on the Monte-Carlo integration method is derived to
quantify the uncertainty of the parameters and predictions. The effectiveness
of the proposed Bayesian approach is demonstrated on several linear and
nonlinear system identification benchmarks by achieving good and competitive
simulation accuracy. The code to reproduce the experimental results is
open-sourced and available online.
