NetScore: Towards Universal Metrics for Large-scale Performance Analysis
  of Deep Neural Networks for Practical On-Device Edge Usage
  Much of the focus in the design of deep neural networks has been on improving
accuracy, leading to more powerful yet highly complex network architectures
that are difficult to deploy in practical scenarios, particularly on edge
devices such as mobile and other consumer devices given their high
computational and memory requirements. As a result, there has been a recent
interest in the design of quantitative metrics for evaluating deep neural
networks that accounts for more than just model accuracy as the sole indicator
of network performance. In this study, we continue the conversation towards
universal metrics for evaluating the performance of deep neural networks for
practical on-device edge usage. In particular, we propose a new balanced metric
called NetScore, which is designed specifically to provide a quantitative
assessment of the balance between accuracy, computational complexity, and
network architecture complexity of a deep neural network, which is important
for on-device edge operation. In what is one of the largest comparative
analysis between deep neural networks in literature, the NetScore metric, the
top-1 accuracy metric, and the popular information density metric were compared
across a diverse set of 60 different deep convolutional neural networks for
image classification on the ImageNet Large Scale Visual Recognition Challenge
(ILSVRC 2012) dataset. The evaluation results across these three metrics for
this diverse set of networks are presented in this study to act as a reference
guide for practitioners in the field. The proposed NetScore metric, along with
the other tested metrics, are by no means perfect, but the hope is to push the
conversation towards better universal metrics for evaluating deep neural
networks for use in practical on-device edge scenarios to help guide
practitioners in model design for such scenarios.
