Play to Grade: Testing Coding Games as Classifying Markov Decision
  Process
  Contemporary coding education often presents students with the task of
developing programs that have user interaction and complex dynamic systems,
such as mouse based games. While pedagogically compelling, there are no
contemporary autonomous methods for providing feedback. Notably, interactive
programs are impossible to grade by traditional unit tests. In this paper we
formalize the challenge of providing feedback to interactive programs as a task
of classifying Markov Decision Processes (MDPs). Each student's program fully
specifies an MDP where the agent needs to operate and decide, under reasonable
generalization, if the dynamics and reward model of the input MDP should be
categorized as correct or broken. We demonstrate that by designing a
cooperative objective between an agent and an autoregressive model, we can use
the agent to sample differential trajectories from the input MDP that allows a
classifier to determine membership: Play to Grade. Our method enables an
automatic feedback system for interactive code assignments. We release a
dataset of 711,274 anonymized student submissions to a single assignment with
hand-coded bug labels to support future research.
