Generative Grading: Near Human-level Accuracy for Automated Feedback on
  Richly Structured Problems
  Access to high-quality education at scale is limited by the difficulty of
providing student feedback on open-ended assignments in structured domains like
computer programming, graphics, and short response questions. This problem has
proven to be exceptionally difficult: for humans, it requires large amounts of
manual work, and for computers, until recently, achieving anything near
human-level accuracy has been unattainable. In this paper, we present
generative grading: a novel computational approach for providing feedback at
scale that is capable of accurately grading student work and providing nuanced,
interpretable feedback. Our approach uses generative descriptions of student
cognition, written as probabilistic programs, to synthesise millions of
labelled example solutions to a problem; we then learn to infer feedback for
real student solutions based on this cognitive model.
  We apply our methods to three settings. In block-based coding, we achieve a
50% improvement upon the previous best results for feedback, achieving
super-human accuracy. In two other widely different domains -- graphical tasks
and short text answers -- we achieve major improvement over the previous state
of the art by about 4x and 1.5x respectively, approaching human accuracy. In a
real classroom, we ran an experiment where we used our system to augment human
graders, yielding doubled grading accuracy while halving grading time.
