LETI: Latency Estimation Tool and Investigation of Neural Networks
  inference on Mobile GPU
  A lot of deep learning applications are desired to be run on mobile devices.
Both accuracy and inference time are meaningful for a lot of them. While the
number of FLOPs is usually used as a proxy for neural network latency, it may
be not the best choice. In order to obtain a better approximation of latency,
research community uses look-up tables of all possible layers for latency
calculation for the final prediction of the inference on mobile CPU. It
requires only a small number of experiments. Unfortunately, on mobile GPU this
method is not applicable in a straight-forward way and shows low precision. In
this work, we consider latency approximation on mobile GPU as a data and
hardware-specific problem. Our main goal is to construct a convenient latency
estimation tool for investigation(LETI) of neural network inference and
building robust and accurate latency prediction models for each specific task.
To achieve this goal, we build open-source tools which provide a convenient way
to conduct massive experiments on different target devices focusing on mobile
GPU. After evaluation of the dataset, we learn the regression model on
experimental data and use it for future latency prediction and analysis. We
experimentally demonstrate the applicability of such an approach on a subset of
popular NAS-Benchmark 101 dataset and also evaluate the most popular neural
network architectures for two mobile GPUs. As a result, we construct latency
prediction model with good precision on the target evaluation subset. We
consider LETI as a useful tool for neural architecture search or massive
latency evaluation. The project is available at https://github.com/leti-ai
