Interpretable feature subset selection: A Shapley value based approach
  For feature selection and related problems, we introduce the notion of
classification game, a cooperative game, with features as players and hinge
loss based characteristic function and relate a feature's contribution to
Shapley value based error apportioning (SVEA) of total training error. Our
major contribution is ($\star$) to show that for any dataset the threshold 0 on
SVEA value identifies feature subset whose joint interactions for label
prediction is significant or those features that span a subspace where the data
is predominantly lying. In addition, our scheme ($\star$) identifies the
features on which Bayes classifier doesn't depend but any surrogate loss
function based finite sample classifier does; this contributes to the excess
$0$-$1$ risk of such a classifier, ($\star$) estimates unknown true hinge risk
of a feature, and ($\star$) relate the stability property of an allocation and
negative valued SVEA by designing the analogue of core of classification game.
Due to Shapley value's computationally expensive nature, we build on a known
Monte Carlo based approximation algorithm that computes characteristic function
(Linear Programs) only when needed. We address the potential sample bias
problem in feature selection by providing interval estimates for SVEA values
obtained from multiple sub-samples. We illustrate all the above aspects on
various synthetic and real datasets and show that our scheme achieves better
results than existing recursive feature elimination technique and ReliefF in
most cases. Our theoretically grounded classification game in terms of well
defined characteristic function offers interpretability (which we formalize in
terms of final task) and explainability of our framework, including
identification of important features.
