Uncertainty aware audiovisual activity recognition using deep Bayesian
  variational inference
  Deep neural networks (DNNs) provide state-of-the-art results for a multitude
of applications, but the approaches using DNNs for multimodal audiovisual
applications do not consider predictive uncertainty associated with individual
modalities. Bayesian deep learning methods provide principled confidence and
quantify predictive uncertainty. Our contribution in this work is to propose an
uncertainty aware multimodal Bayesian fusion framework for activity
recognition. We demonstrate a novel approach that combines deterministic and
variational layers to scale Bayesian DNNs to deeper architectures. Our
experiments using in- and out-of-distribution samples selected from a subset of
Moments-in-Time (MiT) dataset show a more reliable confidence measure as
compared to the non-Bayesian baseline and the Monte Carlo dropout (MC dropout)
approximate Bayesian inference. We also demonstrate the uncertainty estimates
obtained from the proposed framework can identify out-of-distribution data on
the UCF101 and MiT datasets. In the multimodal setting, the proposed framework
improved precision-recall AUC by 10.2% on the subset of MiT dataset as compared
to non-Bayesian baseline.
