A Conversational Paradigm for Program Synthesis
  Program synthesis strives to generate a computer program as a solution to a
given problem specification. We propose a conversational program synthesis
approach via large language models, which addresses the challenges of searching
over a vast program space and user intent specification faced in prior
approaches. Our new approach casts the process of writing a specification and
program as a multi-turn conversation between a user and a system. It treats
program synthesis as a sequence prediction problem, in which the specification
is expressed in natural language and the desired program is conditionally
sampled. We train a family of large language models, called CodeGen, on natural
language and programming language data. With weak supervision in the data and
the scaling up of data size and model size, conversational capacities emerge
from the simple autoregressive language modeling. To study the model behavior
on conversational program synthesis, we develop a multi-turn programming
benchmark (MTPB), where solving each problem requires multi-step synthesis via
multi-turn conversation between the user and the model. Our findings show the
emergence of conversational capabilities and the effectiveness of the proposed
conversational program synthesis paradigm. In addition, our model CodeGen (with
up to 16B parameters trained on TPU-v4) outperforms OpenAI's Codex on the
HumanEval benchmark. We make the training library JaxFormer including
checkpoints available as open source contribution:
https://github.com/salesforce/CodeGen.
