A Tensor Compiler for Unified Machine Learning Prediction Serving
  Machine Learning (ML) adoption in the enterprise requires simpler and more
efficient software infrastructure---the bespoke solutions typical in large web
companies are simply untenable. Model scoring, the process of obtaining
predictions from a trained model over new data, is a primary contributor to
infrastructure complexity and cost as models are trained once but used many
times. In this paper we propose HUMMINGBIRD, a novel approach to model scoring,
which compiles featurization operators and traditional ML models (e.g.,
decision trees) into a small set of tensor operations. This approach inherently
reduces infrastructure complexity and directly leverages existing investments
in Neural Network compilers and runtimes to generate efficient computations for
both CPU and hardware accelerators. Our performance results are intriguing:
despite replacing imperative computations (e.g., tree traversals) with tensor
computation abstractions, HUMMINGBIRD is competitive and often outperforms
hand-crafted kernels on micro-benchmarks on both CPU and GPU, while enabling
seamless end-to-end acceleration of ML pipelines. We have released HUMMINGBIRD
as open source.
