CodeSearchNet Challenge: Evaluating the State of Semantic Code Search
  Semantic code search is the task of retrieving relevant code given a natural
language query. While related to other information retrieval tasks, it requires
bridging the gap between the language used in code (often abbreviated and
highly technical) and natural language more suitable to describe vague concepts
and ideas.
  To enable evaluation of progress on code search, we are releasing the
CodeSearchNet Corpus and are presenting the CodeSearchNet Challenge, which
consists of 99 natural language queries with about 4k expert relevance
annotations of likely results from CodeSearchNet Corpus. The corpus contains
about 6 million functions from open-source code spanning six programming
languages (Go, Java, JavaScript, PHP, Python, and Ruby). The CodeSearchNet
Corpus also contains automatically generated query-like natural language for 2
million functions, obtained from mechanically scraping and preprocessing
associated function documentation. In this article, we describe the methodology
used to obtain the corpus and expert labels, as well as a number of simple
baseline solutions for the task.
  We hope that CodeSearchNet Challenge encourages researchers and practitioners
to study this interesting task further and will host a competition and
leaderboard to track the progress on the challenge. We are also keen on
extending CodeSearchNet Challenge to more queries and programming languages in
the future.
