Interactive Prior Elicitation of Feature Similarities for Small Sample
  Size Prediction
  Regression under the "small $n$, large $p$" conditions, of small sample size
$n$ and large number of features $p$ in the learning data set, is a recurring
setting in which learning from data is difficult. With prior knowledge about
relationships of the features, $p$ can effectively be reduced, but explicating
such prior knowledge is difficult for experts. In this paper we introduce a new
method for eliciting expert prior knowledge about the similarity of the roles
of features in the prediction task. The key idea is to use an interactive
multidimensional-scaling (MDS) type scatterplot display of the features to
elicit the similarity relationships, and then use the elicited relationships in
the prior distribution of prediction parameters. Specifically, for learning to
predict a target variable with Bayesian linear regression, the feature
relationships are used to construct a Gaussian prior with a full covariance
matrix for the regression coefficients. Evaluation of our method in experiments
with simulated and real users on text data confirm that prior elicitation of
feature similarities improves prediction accuracy. Furthermore, elicitation
with an interactive scatterplot display outperforms straightforward elicitation
where the users choose feature pairs from a feature list.
