Robust Importance Sampling for Error Estimation in the Context of
  Optimal Bayesian Transfer Learning
  Classification has been a major task for building intelligent systems as it
enables decision-making under uncertainty. Classifier design aims at building
models from training data for representing feature-label distributions--either
explicitly or implicitly. In many scientific or clinical settings, training
data are typically limited, which makes designing accurate classifiers and
evaluating their classification error extremely challenging. While transfer
learning (TL) can alleviate this issue by incorporating data from relevant
source domains to improve learning in a different target domain, it has
received little attention for performance assessment, notably in error
estimation. In this paper, we fill this gap by investigating knowledge
transferability in the context of classification error estimation within a
Bayesian paradigm. We introduce a novel class of Bayesian minimum mean-square
error (MMSE) estimators for optimal Bayesian transfer learning (OBTL), which
enables rigorous evaluation of classification error under uncertainty in a
small-sample setting. Using Monte Carlo importance sampling, we employ the
proposed estimator to evaluate the classification accuracy of a broad family of
classifiers that span diverse learning capabilities. Experimental results based
on both synthetic data as well as real-world RNA sequencing (RNA-seq) data show
that our proposed OBTL error estimation scheme clearly outperforms standard
error estimators, especially in a small-sample setting, by tapping into the
data from other relevant domains.
