VarCLR: Variable Semantic Representation Pre-training via Contrastive
  Learning
  Variable names are critical for conveying intended program behavior. Machine
learning-based program analysis methods use variable name representations for a
wide range of tasks, such as suggesting new variable names and bug detection.
Ideally, such methods could capture semantic relationships between names beyond
syntactic similarity, e.g., the fact that the names average and mean are
similar. Unfortunately, previous work has found that even the best of previous
representation approaches primarily capture relatedness (whether two variables
are linked at all), rather than similarity (whether they actually have the same
meaning).
  We propose VarCLR, a new approach for learning semantic representations of
variable names that effectively captures variable similarity in this stricter
sense. We observe that this problem is an excellent fit for contrastive
learning, which aims to minimize the distance between explicitly similar
inputs, while maximizing the distance between dissimilar inputs. This requires
labeled training data, and thus we construct a novel, weakly-supervised
variable renaming dataset mined from GitHub edits. We show that VarCLR enables
the effective application of sophisticated, general-purpose language models
like BERT, to variable name representation and thus also to related downstream
tasks like variable name similarity search or spelling correction. VarCLR
produces models that significantly outperform the state-of-the-art on IdBench,
an existing benchmark that explicitly captures variable similarity (as distinct
from relatedness). Finally, we contribute a release of all data, code, and
pre-trained models, aiming to provide a drop-in replacement for variable
representations used in either existing or future program analyses that rely on
variable names.
