InferBench: Understanding Deep Learning Inference Serving with an
  Automatic Benchmarking System
  Deep learning (DL) models have become core modules for many applications.
However, deploying these models without careful performance benchmarking that
considers both hardware and software's impact often leads to poor service and
costly operational expenditure. To facilitate DL models' deployment, we
implement an automatic and comprehensive benchmark system for DL developers. To
accomplish benchmark-related tasks, the developers only need to prepare a
configuration file consisting of a few lines of code. Our system, deployed to a
leader server in DL clusters, will dispatch users' benchmark jobs to follower
workers. Next, the corresponding requests, workload, and even models can be
generated automatically by the system to conduct DL serving benchmarks.
Finally, developers can leverage many analysis tools and models in our system
to gain insights into the trade-offs of different system configurations. In
addition, a two-tier scheduler is incorporated to avoid unnecessary
interference and improve average job compilation time by up to 1.43x
(equivalent of 30\% reduction). Our system design follows the best practice in
DL clusters operations to expedite day-to-day DL service evaluation efforts by
the developers. We conduct many benchmark experiments to provide in-depth and
comprehensive evaluations. We believe these results are of great values as
guidelines for DL service configuration and resource allocation.
