MicroRec: Efficient Recommendation Inference by Hardware and Data
  Structure Solutions
  Deep neural networks are widely used in personalized recommendation systems.
Unlike regular DNN inference workloads, recommendation inference is
memory-bound due to the many random memory accesses needed to lookup the
embedding tables. The inference is also heavily constrained in terms of latency
because producing a recommendation for a user must be done in about tens of
milliseconds. In this paper, we propose MicroRec, a high-performance inference
engine for recommendation systems. MicroRec accelerates recommendation
inference by (1) redesigning the data structures involved in the embeddings to
reduce the number of lookups needed and (2) taking advantage of the
availability of High-Bandwidth Memory (HBM) in FPGA accelerators to tackle the
latency by enabling parallel lookups. We have implemented the resulting design
on an FPGA board including the embedding lookup step as well as the complete
inference process. Compared to the optimized CPU baseline (16 vCPU,
AVX2-enabled), MicroRec achieves 13.8~14.7x speedup on embedding lookup alone
and 2.5$~5.4x speedup for the entire recommendation inference in terms of
throughput. As for latency, CPU-based engines needs milliseconds for inferring
a recommendation while MicroRec only takes microseconds, a significant
advantage in real-time recommendation systems.
