How effective can simple ordinal peer grading be?
  Ordinal peer grading has been proposed as a simple and scalable solution for
computing reliable information about student performance in massive open online
courses. The idea is to outsource the grading task to the students themselves
as follows. After the end of an exam, each student is asked to rank -- in terms
of quality -- a bundle of exam papers by fellow students. An aggregation rule
then combines the individual rankings into a global one that contains all
students. We define a broad class of simple aggregation rules, which we call
type-ordering aggregation rules, and present a theoretical framework for
assessing their effectiveness. When statistical information about the grading
behaviour of students is available (in terms of a noise matrix that
characterizes the grading behaviour of the average student from a student
population), the framework can be used to compute the optimal rule from this
class with respect to a series of performance objectives that compare the
ranking returned by the aggregation rule to the underlying ground truth
ranking. For example, a natural rule known as Borda is proved to be optimal
when students grade correctly. In addition, we present extensive simulations
that validate our theory and prove it to be extremely accurate in predicting
the performance of aggregation rules even when only rough information about
grading behaviour (i.e., an approximation of the noise matrix) is available.
Both in the application of our theoretical framework and in our simulations, we
exploit data about grading behaviour of students that have been extracted from
two field experiments in the University of Patras.
