How Much Data Analytics is Enough? The ROI of Machine Learning
  Classification and its Application to Requirements Dependency Classification
  Machine Learning (ML) can substantially improve the efficiency and
effectiveness of organizations and is widely used for different purposes within
Software Engineering. However, the selection and implementation of ML
techniques rely almost exclusively on accuracy criteria. Thus, for
organizations wishing to realize the benefits of ML investments, this narrow
approach ignores crucial considerations around the anticipated costs of the ML
activities across the ML lifecycle, while failing to account for the benefits
that are likely to accrue from the proposed activity. We present findings for
an approach that addresses this gap by enhancing the accuracy criterion with
return on investment (ROI) considerations. Specifically, we analyze the
performance of the two state-of-the-art ML techniques: Random Forest and
Bidirectional Encoder Representations from Transformers (BERT), based on
accuracy and ROI for two publicly available data sets. Specifically, we compare
decision-making on requirements dependency extraction (i) exclusively based on
accuracy and (ii) extended to include ROI analysis. As a result, we propose
recommendations for selecting ML classification techniques based on the degree
of training data used. Our findings indicate that considering ROI as additional
criteria can drastically influence ML selection when compared to decisions
based on accuracy as the sole criterion
