Design and Analysis of the NIPS 2016 Review Process
  Neural Information Processing Systems (NIPS) is a top-tier annual conference
in machine learning. The 2016 edition of the conference comprised more than
2,400 paper submissions, 3,000 reviewers, and 8,000 attendees. This represents
a growth of nearly 40% in terms of submissions, 96% in terms of reviewers, and
over 100% in terms of attendees as compared to the previous year. The massive
scale as well as rapid growth of the conference calls for a thorough quality
assessment of the peer-review process and novel means of improvement. In this
paper, we analyze several aspects of the data collected during the review
process, including an experiment investigating the efficacy of collecting
ordinal rankings from reviewers. Our goal is to check the soundness of the
review process, and provide insights that may be useful in the design of the
review process of subsequent conferences.
