Your 2 is My 1, Your 3 is My 9: Handling Arbitrary Miscalibrations in
  Ratings
  Cardinal scores (numeric ratings) collected from people are well known to
suffer from miscalibrations. A popular approach to address this issue is to
assume simplistic models of miscalibration (such as linear biases) to de-bias
the scores. This approach, however, often fares poorly because people's
miscalibrations are typically far more complex and not well understood. In the
absence of simplifying assumptions on the miscalibration, it is widely believed
by the crowdsourcing community that the only useful information in the cardinal
scores is the induced ranking. In this paper, inspired by the framework of
Stein's shrinkage, empirical Bayes, and the classic two-envelope problem, we
contest this widespread belief. Specifically, we consider cardinal scores with
arbitrary (or even adversarially chosen) miscalibrations which are only
required to be consistent with the induced ranking. We design estimators which
despite making no assumptions on the miscalibration, strictly and uniformly
outperform all possible estimators that rely on only the ranking. Our
estimators are flexible in that they can be used as a plug-in for a variety of
applications, and we provide a proof-of-concept for A/B testing and ranking.
Our results thus provide novel insights in the eternal debate between cardinal
and ordinal data.
