Beyond the Hype: A Real-World Evaluation of the Impact and Cost of
  Machine Learning-Based Malware Detection
  In this paper, we present a scientific evaluation of four market-leading
malware detection tools to assist an organization with two primary questions:
To what extent do ML-based tools accurately classify previously- and
never-before-seen files? Is it worth purchasing a network-level malware
detector? To identify weaknesses, we tested each tool against 3,536 total files
(2,554 or 72\% malicious, 982 or 28\% benign) of a variety of file types,
including hundreds of malicious zero-days, polyglots, and APT-style files,
delivered on multiple protocols. We present statistical results on detection
time and accuracy, consider complementary analysis (using multiple tools
together), and provide two novel applications of the recent cost-benefit
evaluation procedure of Iannacone \& Bridges. While the ML-based tools are more
effective at detecting zero-day files and executables, the signature-based tool
may still be an overall better option. Both network-based tools provide
substantial (simulated) savings when paired with either host tool, yet both
show poor detection rates on protocols other than HTTP or SMTP. Our results
show that all four tools have near-perfect precision but alarmingly low recall,
especially on file types other than executables and office files -- 37% of
malware tested, including all polyglot files, were undetected. Priorities for
researchers and takeaways for end users are given.
