Beyond Accuracy: ROI-driven Data Analytics of Empirical Data
  This vision paper demonstrates that it is crucial to consider
Return-on-Investment (ROI) when performing Data Analytics. Decisions on "How
much analytics is needed"? are hard to answer. ROI could guide for decision
support on the What?, How?, and How Much? analytics for a given problem.
Method: The proposed conceptual framework is validated through two empirical
studies that focus on requirements dependencies extraction in the Mozilla
Firefox project. The two case studies are (i) Evaluation of fine-tuned BERT
against Naive Bayes and Random Forest machine learners for binary dependency
classification and (ii) Active Learning against passive Learning (random
sampling) for REQUIRES dependency extraction. For both the cases, their
analysis investment (cost) is estimated, and the achievable benefit from DA is
predicted, to determine a break-even point of the investigation. Results: For
the first study, fine-tuned BERT performed superior to the Random Forest,
provided that more than 40% of training data is available. For the second,
Active Learning achieved higher F1 accuracy within fewer iterations and higher
ROI compared to Baseline (Random sampling based RF classifier). In both the
studies, estimate on, How much analysis likely would pay off for the invested
efforts?, was indicated by the break-even point. Conclusions: Decisions for the
depth and breadth of DA of empirical data should not be made solely based on
the accuracy measures. Since ROI-driven Data Analytics provides a simple yet
effective direction to discover when to stop further investigation while
considering the cost and value of the various types of analysis, it helps to
avoid over-analyzing empirical data.
