Why Do Machine Learning Practitioners Still Use Manual Tuning? A
  Qualitative Study
  Current advanced hyperparameter optimization (HPO) methods, such as Bayesian
optimization, have high sampling efficiency and facilitate replicability.
Nonetheless, machine learning (ML) practitioners (e.g., engineers, scientists)
mostly apply less advanced HPO methods, which can increase resource consumption
during HPO or lead to underoptimized ML models. Therefore, we suspect that
practitioners choose their HPO method to achieve different goals, such as
decrease practitioner effort and target audience compliance. To develop HPO
methods that align with such goals, the reasons why practitioners decide for
specific HPO methods must be unveiled and thoroughly understood. Because
qualitative research is most suitable to uncover such reasons and find
potential explanations for them, we conducted semi-structured interviews to
explain why practitioners choose different HPO methods. The interviews revealed
six principal practitioner goals (e.g., increasing model comprehension), and
eleven key factors that impact decisions for HPO methods (e.g., available
computing resources). We deepen the understanding about why practitioners
decide for different HPO methods and outline recommendations for improvements
of HPO methods by aligning them with practitioner goals.
