Choose Your Programming Copilot: A Comparison of the Program Synthesis
  Performance of GitHub Copilot and Genetic Programming
  GitHub Copilot, an extension for the Visual Studio Code development
environment powered by the large-scale language model Codex, makes automatic
program synthesis available for software developers. This model has been
extensively studied in the field of deep learning, however, a comparison to
genetic programming, which is also known for its performance in automatic
program synthesis, has not yet been carried out. In this paper, we evaluate
GitHub Copilot on standard program synthesis benchmark problems and compare the
achieved results with those from the genetic programming literature. In
addition, we discuss the performance of both approaches. We find that the
performance of the two approaches on the benchmark problems is quite similar,
however, in comparison to GitHub Copilot, the program synthesis approaches
based on genetic programming are not yet mature enough to support programmers
in practical software development. Genetic programming usually needs a huge
amount of expensive hand-labeled training cases and takes too much time to
generate solutions. Furthermore, source code generated by genetic programming
approaches is often bloated and difficult to understand. For future work on
program synthesis with genetic programming, we suggest researchers to focus on
improving the execution time, readability, and usability.
