A Large Scale Randomized Controlled Trial on Herding in Peer-Review
  Discussions
  Peer review is the backbone of academia and humans constitute a cornerstone
of this process, being responsible for reviewing papers and making the final
acceptance/rejection decisions. Given that human decision making is known to be
susceptible to various cognitive biases, it is important to understand which
(if any) biases are present in the peer-review process and design the pipeline
such that the impact of these biases is minimized. In this work, we focus on
the dynamics of between-reviewers discussions and investigate the presence of
herding behaviour therein. In that, we aim to understand whether reviewers and
more senior decision makers get disproportionately influenced by the first
argument presented in the discussion when (in case of reviewers) they form an
independent opinion about the paper before discussing it with others.
Specifically, in conjunction with the review process of ICML 2020 -- a large,
top tier machine learning conference -- we design and execute a randomized
controlled trial with the goal of testing for the conditional causal effect of
the discussion initiator's opinion on the outcome of a paper.
