Smart at what cost? Characterising Mobile Deep Neural Networks in the
  wild
  With smartphones' omnipresence in people's pockets, Machine Learning (ML) on
mobile is gaining traction as devices become more powerful. With applications
ranging from visual filters to voice assistants, intelligence on mobile comes
in many forms and facets. However, Deep Neural Network (DNN) inference remains
a compute intensive workload, with devices struggling to support intelligence
at the cost of responsiveness.On the one hand, there is significant research on
reducing model runtime requirements and supporting deployment on embedded
devices. On the other hand, the strive to maximise the accuracy of a task is
supported by deeper and wider neural networks, making mobile deployment of
state-of-the-art DNNs a moving target.
  In this paper, we perform the first holistic study of DNN usage in the wild
in an attempt to track deployed models and match how these run on widely
deployed devices. To this end, we analyse over 16k of the most popular apps in
the Google Play Store to characterise their DNN usage and performance across
devices of different capabilities, both across tiers and generations.
Simultaneously, we measure the models' energy footprint, as a core cost
dimension of any mobile deployment. To streamline the process, we have
developed gaugeNN, a tool that automates the deployment, measurement and
analysis of DNNs on devices, with support for different frameworks and
platforms. Results from our experience study paint the landscape of deep
learning deployments on smartphones and indicate their popularity across app
developers. Furthermore, our study shows the gap between bespoke techniques and
real-world deployments and the need for optimised deployment of deep learning
models in a highly dynamic and heterogeneous ecosystem.
