Real World Large Scale Recommendation Systems Reproducibility and Smooth
  Activations
  Real world recommendation systems influence a constantly growing set of
domains. With deep networks, that now drive such systems, recommendations have
been more relevant to the user's interests and tasks. However, they may not
always be reproducible even if produced by the same system for the same user,
recommendation sequence, request, or query. This problem received almost no
attention in academic publications, but is, in fact, very realistic and
critical in real production systems. We consider reproducibility of real large
scale deep models, whose predictions determine such recommendations. We
demonstrate that the celebrated Rectified Linear Unit (ReLU) activation, used
in deep models, can be a major contributor to irreproducibility. We propose the
use of smooth activations to improve recommendation reproducibility. We
describe a novel family of smooth activations; Smooth ReLU (SmeLU), designed to
improve reproducibility with mathematical simplicity, with potentially cheaper
implementation. SmeLU is a member of a wider family of smooth activations.
While other techniques that improve reproducibility in real systems usually
come at accuracy costs, smooth activations not only improve reproducibility,
but can even give accuracy gains. We report metrics from real systems in which
we were able to productionalize SmeLU with substantial reproducibility gains
and better accuracy-reproducibility trade-offs. These include
click-through-rate (CTR) prediction systems, content, and application
recommendation systems.
