PREMA: A Predictive Multi-task Scheduling Algorithm For Preemptible
  Neural Processing Units
  To amortize cost, cloud vendors providing DNN acceleration as a service to
end-users employ consolidation and virtualization to share the underlying
resources among multiple DNN service requests. This paper makes a case for a
"preemptible" neural processing unit (NPU) and a "predictive" multi-task
scheduler to meet the latency demands of high-priority inference while
maintaining high throughput. We evaluate both the mechanisms that enable NPUs
to be preemptible and the policies that utilize them to meet scheduling
objectives. We show that preemptive NPU multi-tasking can achieve an average
7.8x, 1.4x, and 4.8x improvement in latency, throughput, and SLA satisfaction,
respectively.
