Demystifying a Dark Art: Understanding Real-World Machine Learning Model
  Development
  It is well-known that the process of developing machine learning (ML)
workflows is a dark-art; even experts struggle to find an optimal workflow
leading to a high accuracy model. Users currently rely on empirical
trial-and-error to obtain their own set of battle-tested guidelines to inform
their modeling decisions. In this study, we aim to demystify this dark art by
understanding how people iterate on ML workflows in practice. We analyze over
475k user-generated workflows on OpenML, an open-source platform for tracking
and sharing ML workflows. We find that users often adopt a manual, automated,
or mixed approach when iterating on their workflows. We observe that manual
approaches result in fewer wasted iterations compared to automated approaches.
Yet, automated approaches often involve more preprocessing and hyperparameter
options explored, resulting in higher performance overall--suggesting potential
benefits for a human-in-the-loop ML system that appropriately recommends a
clever combination of the two strategies.
