ProtoTransformer: A Meta-Learning Approach to Providing Student Feedback
  High-quality computer science education is limited by the difficulty of
providing instructor feedback to students at scale. While this feedback could
in principle be automated, supervised approaches to predicting the correct
feedback are bottlenecked by the intractability of annotating large quantities
of student code. In this paper, we instead frame the problem of providing
feedback as few-shot classification, where a meta-learner adapts to give
feedback to student code on a new programming question from just a few examples
annotated by instructors. Because data for meta-training is limited, we propose
a number of amendments to the typical few-shot learning framework, including
task augmentation to create synthetic tasks, and additional side information to
build stronger priors about each task. These additions are combined with a
transformer architecture to embed discrete sequences (e.g. code) to a
prototypical representation of a feedback class label. On a suite of few-shot
natural language processing tasks, we match or outperform state-of-the-art
performance. Then, on a collection of student solutions to exam questions from
an introductory university course, we show that our approach reaches an average
precision of 88% on unseen questions, surpassing the 82% precision of teaching
assistants. Our approach was successfully deployed to deliver feedback to
16,000 student exam-solutions in a programming course offered by a tier 1
university. This is, to the best of our knowledge, the first successful
deployment of a machine learning based feedback to open-ended student code.
