Production Ranking Systems: A Review
  The problem of ranking is a multi-billion dollar problem. In this paper we
present an overview of several production quality ranking systems. We show that
due to conflicting goals of employing the most effective machine learning
models and responding to users in real time, ranking systems have evolved into
a system of systems, where each subsystem can be viewed as a component layer.
We view these layers as being data processing, representation learning,
candidate selection and online inference. Each layer employs different
algorithms and tools, with every end-to-end ranking system spanning multiple
architectures. Our goal is to familiarize the general audience with a working
knowledge of ranking at scale, the tools and algorithms employed and the
challenges introduced by adopting a layered approach.
