Understanding the Nature of System-Related Issues in Machine Learning
  Frameworks: An Exploratory Study
  Modern systems are built using development frameworks. These frameworks have
a major impact on how the resulting system executes, how configurations are
managed, how it is tested, and how and where it is deployed. Machine learning
(ML) frameworks and the systems developed using them differ greatly from
traditional frameworks. Naturally, the issues that manifest in such frameworks
may differ as well---as may the behavior of developers addressing those issues.
We are interested in characterizing the system-related issues---issues
impacting performance, memory and resource usage, and other quality
attributes---that emerge in ML frameworks, and how they differ from those in
traditional frameworks. We have conducted a moderate-scale exploratory study
analyzing real-world system-related issues from 10 popular machine learning
frameworks. Our findings offer implications for the development of machine
learning systems, including differences in the frequency of occurrence of
certain issue types, observations regarding the impact of debate and time on
issue correction, and differences in the specialization of developers. We hope
that this exploratory study will enable developers to improve their
expectations, plan for risk, and allocate resources accordingly when making use
of the tools provided by these frameworks to develop ML-based systems.
