Benchmarking Uncertainty Quantification on Biosignal Classification
  Tasks under Dataset Shift
  A biosignal is a signal that can be continuously measured from human bodies,
such as respiratory sounds, heart activity (ECG), brain waves (EEG), etc, based
on which, machine learning models have been developed with very promising
performance for automatic disease detection and health status monitoring.
However, dataset shift, i.e., data distribution of inference varies from the
distribution of the training, is not uncommon for real biosignal-based
applications. To improve the robustness, probabilistic models with uncertainty
quantification are adapted to capture how reliable a prediction is. Yet,
assessing the quality of the estimated uncertainty remains a challenge. In this
work, we propose a framework to evaluate the capability of the estimated
uncertainty in capturing different types of biosignal dataset shifts with
various degrees. In particular, we use three classification tasks based on
respiratory sounds and electrocardiography signals to benchmark five
representative uncertainty quantification methods. Extensive experiments show
that, although Ensemble and Bayesian models could provide relatively better
uncertainty estimations under dataset shifts, all tested models fail to meet
the promise in trustworthy prediction and model calibration. Our work paves the
way for a comprehensive evaluation for any newly developed biosignal
classifiers.
