CORAL: COde RepresentAtion Learning with Weakly-Supervised Transformers
  for Analyzing Data Analysis
  Large scale analysis of source code, and in particular scientific source
code, holds the promise of better understanding the data science process,
identifying analytical best practices, and providing insights to the builders
of scientific toolkits. However, large corpora have remained unanalyzed in
depth, as descriptive labels are absent and require expert domain knowledge to
generate. We propose a novel weakly supervised transformer-based architecture
for computing joint representations of code from both abstract syntax trees and
surrounding natural language comments. We then evaluate the model on a new
classification task for labeling computational notebook cells as stages in the
data analysis process from data import to wrangling, exploration, modeling, and
evaluation. We show that our model, leveraging only easily-available weak
supervision, achieves a 38% increase in accuracy over expert-supplied
heuristics and outperforms a suite of baselines. Our model enables us to
examine a set of 118,000 Jupyter Notebooks to uncover common data analysis
patterns. Focusing on notebooks with relationships to academic articles, we
conduct the largest ever study of scientific code and find that notebook
composition correlates with the citation count of corresponding papers.
