A Note on Latency Variability of Deep Neural Networks for Mobile
  Inference
  Running deep neural network (DNN) inference on mobile devices, i.e., mobile
inference, has become a growing trend, making inference less dependent on
network connections and keeping private data locally. The prior studies on
optimizing DNNs for mobile inference typically focus on the metric of average
inference latency, thus implicitly assuming that mobile inference exhibits
little latency variability. In this note, we conduct a preliminary measurement
study on the latency variability of DNNs for mobile inference. We show that the
inference latency variability can become quite significant in the presence of
CPU resource contention. More interestingly, unlike the common belief that the
relative performance superiority of DNNs on one device can carry over to
another device and/or another level of resource contention, we highlight that a
DNN model with a better latency performance than another model can become
outperformed by the other model when resource contention be more severe or
running on another device. Thus, when optimizing DNN models for mobile
inference, only measuring the average latency may not be adequate; instead,
latency variability under various conditions should be accounted for, including
but not limited to different devices and different levels of CPU resource
contention considered in this note.
