Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS
  Experiment
  In this paper we revisit the 2014 NeurIPS experiment that examined
inconsistency in conference peer review. We determine that 50\% of the
variation in reviewer quality scores was subjective in origin. Further, with
seven years passing since the experiment we find that for \emph{accepted}
papers, there is no correlation between quality scores and impact of the paper
as measured as a function of citation count. We trace the fate of rejected
papers, recovering where these papers were eventually published. For these
papers we find a correlation between quality scores and impact. We conclude
that the reviewing process for the 2014 conference was good for identifying
poor papers, but poor for identifying good papers. We give some suggestions for
improving the reviewing process but also warn against removing the subjective
element. Finally, we suggest that the real conclusion of the experiment is that
the community should place less onus on the notion of `top-tier conference
publications' when assessing the quality of individual researchers. For NeurIPS
2021, the PCs are repeating the experiment, as well as conducting new ones.
