Low-Precision Hardware Architectures Meet Recommendation Model Inference
  at Scale
  Tremendous success of machine learning (ML) and the unabated growth in ML
model complexity motivated many ML-specific designs in both CPU and accelerator
architectures to speed up the model inference. While these architectures are
diverse, highly optimized low-precision arithmetic is a component shared by
most. Impressive compute throughputs are indeed often exhibited by these
architectures on benchmark ML models. Nevertheless, production models such as
recommendation systems important to Facebook's personalization services are
demanding and complex: These systems must serve billions of users per month
responsively with low latency while maintaining high prediction accuracy,
notwithstanding computations with many tens of billions parameters per
inference. Do these low-precision architectures work well with our production
recommendation systems? They do. But not without significant effort. We share
in this paper our search strategies to adapt reference recommendation models to
low-precision hardware, our optimization of low-precision compute kernels, and
the design and development of tool chain so as to maintain our models' accuracy
throughout their lifespan during which topic trends and users' interests
inevitably evolve. Practicing these low-precision technologies helped us save
datacenter capacities while deploying models with up to 5X complexity that
would otherwise not be deployed on traditional general-purpose CPUs. We believe
these lessons from the trenches promote better co-design between hardware
architecture and software engineering and advance the state of the art of ML in
industry.
