The Adverse Effects of Code Duplication in Machine Learning Models of
  Code
  The field of big code relies on mining large corpora of code to perform some
learning task. A significant threat to this approach has been recently
identified by Lopes et al. (2017) who found a large amount of near-duplicate
code on GitHub. However, the impact of code duplication has not been noticed by
researchers devising machine learning models for source code. In this work, we
explore the effects of code duplication on machine learning models showing that
reported performance metrics are sometimes inflated by up to 100% when testing
on duplicated code corpora compared to the performance on de-duplicated corpora
which more accurately represent how machine learning models of code are used by
software engineers. We present a duplication index for widely used datasets,
list best practices for collecting code corpora and evaluating machine learning
models on them. Finally, we release tools to help the community avoid this
problem in future research.
