Comparison and Benchmarking of AI Models and Frameworks on Mobile
  Devices
  Due to increasing amounts of data and compute resources, deep learning
achieves many successes in various domains. The application of deep learning on
the mobile and embedded devices is taken more and more attentions, benchmarking
and ranking the AI abilities of mobile and embedded devices becomes an urgent
problem to be solved. Considering the model diversity and framework diversity,
we propose a benchmark suite, AIoTBench, which focuses on the evaluation of the
inference abilities of mobile and embedded devices. AIoTBench covers three
typical heavy-weight networks: ResNet50, InceptionV3, DenseNet121, as well as
three light-weight networks: SqueezeNet, MobileNetV2, MnasNet. Each network is
implemented by three frameworks which are designed for mobile and embedded
devices: Tensorflow Lite, Caffe2, Pytorch Mobile. To compare and rank the AI
capabilities of the devices, we propose two unified metrics as the AI scores:
Valid Images Per Second (VIPS) and Valid FLOPs Per Second (VOPS). Currently, we
have compared and ranked 5 mobile devices using our benchmark. This list will
be extended and updated soon after.
