Implications on Feature Detection when using the Benefit-Cost Ratio
  In many practical machine learning applications, there are two objectives:
one is to maximize predictive accuracy and the other is to minimize costs of
the resulting model. These costs of individual features may be financial costs,
but can also refer to other aspects, like for example evaluation time. Feature
selection addresses both objectives, as it reduces the number of features and
can improve the generalization ability of the model. If costs differ between
features, the feature selection needs to trade-off the individual benefit and
cost of each feature. A popular trade-off choice is the ratio of both, the BCR
(benefit-cost ratio). In this paper we analyze implications of using this
measure with special focus to the ability to distinguish relevant features from
noise. We perform a simulation study for different cost and data settings and
obtain detection rates of relevant features and empirical distributions of the
trade-off ratio. Our simulation study exposed a clear impact of the cost
setting on the detection rate. In situations with large cost differences and
small effect sizes, the BCR missed relevant features and preferred cheap noise
features. We conclude that a trade-off between predictive performance and costs
without a controlling hyperparameter can easily overemphasize very cheap noise
features. While the simple benefit-cost ratio offers an easy solution to
incorporate costs, it is important to be aware of its risks. Avoiding costs
close to 0, rescaling large cost differences, or using a hyperparameter
trade-off are ways to counteract the adverse effects exposed in this paper.
