Software Engineers vs. Machine Learning Algorithms: An Empirical Study
  Assessing Performance and Reuse Tasks
  Several papers have recently contained reports on applying machine learning
(ML) to the automation of software engineering (SE) tasks, such as project
management, modeling and development. However, there appear to be no approaches
comparing how software engineers fare against machine-learning algorithms as
applied to specific software development tasks. Such a comparison is essential
to gain insight into which tasks are better performed by humans and which by
machine learning and how cooperative work or human-in-the-loop processes can be
implemented more effectively. In this paper, we present an empirical study that
compares how software engineers and machine-learning algorithms perform and
reuse tasks. The empirical study involves the synthesis of the control
structure of an autonomous streetlight application. Our approach consists of
four steps. First, we solved the problem using machine learning to determine
specific performance and reuse tasks. Second, we asked software engineers with
different domain knowledge levels to provide a solution to the same tasks.
Third, we compared how software engineers fare against machine-learning
algorithms when accomplishing the performance and reuse tasks based on criteria
such as energy consumption and safety. Finally, we analyzed the results to
understand which tasks are better performed by either humans or algorithms so
that they can work together more effectively. Such an understanding and the
resulting human-in-the-loop approaches, which take into account the strengths
and weaknesses of humans and machine-learning algorithms, are fundamental not
only to provide a basis for cooperative work in support of software
engineering, but also, in other areas.
