LazyBatching: An SLA-aware Batching System for Cloud Machine Learning
  Inference
  In cloud ML inference systems, batching is an essential technique to increase
throughput which helps optimize total-cost-of-ownership. Prior graph batching
combines the individual DNN graphs into a single one, allowing multiple inputs
to be concurrently executed in parallel. We observe that the coarse-grained
graph batching becomes suboptimal in effectively handling the dynamic inference
request traffic, leaving significant performance left on the table. This paper
proposes LazyBatching, an SLA-aware batching system that considers both
scheduling and batching in the granularity of individual graph nodes, rather
than the entire graph for flexible batching. We show that LazyBatching can
intelligently determine the set of nodes that can be efficiently batched
together, achieving an average 15x, 1.5x, and 5.5x improvement than graph
batching in terms of average response time, throughput, and SLA satisfaction,
respectively.
