Zero Shot Learning for Code Education: Rubric Sampling with Deep
  Learning Inference
  In modern computer science education, massive open online courses (MOOCs) log
thousands of hours of data about how students solve coding challenges. Being so
rich in data, these platforms have garnered the interest of the machine
learning community, with many new algorithms attempting to autonomously provide
feedback to help future students learn. But what about those first hundred
thousand students? In most educational contexts (i.e. classrooms), assignments
do not have enough historical data for supervised learning. In this paper, we
introduce a human-in-the-loop "rubric sampling" approach to tackle the "zero
shot" feedback challenge. We are able to provide autonomous feedback for the
first students working on an introductory programming assignment with accuracy
that substantially outperforms data-hungry algorithms and approaches human
level fidelity. Rubric sampling requires minimal teacher effort, can associate
feedback with specific parts of a student's solution and can articulate a
student's misconceptions in the language of the instructor. Deep learning
inference enables rubric sampling to further improve as more assignment
specific student data is acquired. We demonstrate our results on a novel
dataset from Code.org, the world's largest programming education platform.
